System for Accurately and Precisely Representing Image Color Information

ABSTRACT

A method and system for accurate and precise representation of color for still and moving images, particularly sequences of digitized color images. Spectral and/or extended dynamic range information is retained as images are captured, processed, and presented during color adjustment. Using this extra spectral information, various methodologies for further presenting or processing the color within these images can be optimized. Presentation-device independence is achieved not by attempting to discover a device-independent intermediate representation, but rather by deferring the binding and mapping of color representation onto a presentation device until its actual use.

RELATED APPLICATIONS

This application is a non-provisional application which claims priorityof provisional application Ser. No. 61/065,615, filed on Feb. 13, 2008.

This application is a continuation of application Ser. No. 14/072,722,filed Nov. 5, 2013, entitled “System For Accurately and PreciselyRepresenting Image Color Information”, which is a continuation ofapplication Ser. No. 12/319,932, filed Jan. 14, 2009, now U.S. Pat. No.8,593,476, issued on Nov. 26, 2013, entitled “System For Accurately andPrecisely Representing Image Color Information”.

OTHER RELATED APPLICATIONS

“High Quality Wide Range Multi-Layer Image Compression Coding System”,U.S. patent application Ser. No. 11/225,655.

“Film and Video Bi-Directional Color Matching System and Method”, U.S.Patent Application Ser. No. 60/198,890 and Ser. No. 09/648,414.

BACKGROUND

This invention relates to accurate and precise representation of colorfor still and moving images, particularly sequences of digitized colorimages, including digital motion pictures and digital video.

CIE 1931 Standard Colorimetric Observer

During the period 1928 through 1931, J. Guild and W. D. Wright inEngland developed the CIE 1931 XYZ and xyz colorimetric system. Thefundamental elements of this color representation are spectral mappingcurves called x_bar, y_bar, and z_bar. These functions are based upon atransformation from human vision color matching measurements usingamounts of relatively pure red, green, and blue. The red, green, andblue amounts were determined in 2-degree color patch matchingexperiments using a mixture of amounts of these red, green, and blueprimaries to match arbitrary colors (other than red, green, or blue).The functions of the amounts to match arbitrary colors are embodied inspectral functions r_bar, g_bar, and b_bar. These functions havenegative amounts in certain wavelength regions. The x_bar, y_bar, andz_bar spectral color matching functions are a linear transformation ofGuild's and Wrights independent r_bar, g_bar, and b_bar measurements,taking into account the fact that the wavelengths of the red, green, andblue primaries were different between Guild and Wright. The x_bar,y_bar, and z_bar functions have all positive spectral amounts, unliker_bar, g_bar, and b_bar. For general reference, see Color Science,Concepts and Methods, Quantitative Data and Formulae, 2^(nd) Edition1982, by Gunter Wyszecki and Walter Stanley Stiles, John Wiley and Sons;Color Appearance Models, 2^(nd) Edition 2005, by Mark Fairchild, JohnWiley and Sons; and Fundamental Chromaticity Diagram with PsychologicalAxes—Part 1, CIE 170-1:2006 (International Commission on Illumination,Commission Internationale De L'Eclairage, CIE) ISBN 3 901 906 46 0.

A discussion of this original color model work can be found in Wszeckiand Stiles (W&S) in section 3.3 and section 5.5.6. It is significant tonote that this work pre-dated computers and spectral-radiometers (whichdirectly measure spectra digitally). The data for x_bar, y_bar, andz_bar are specified with four decimal digits, and at 5 nm intervalsbetween 380 nm and 780 nm (see W&S Table II (3.3.1), page 736). Thisprecision and accuracy is actually much higher than was implied by theoriginal color examinations performed by Guild and Wright. Although thex_bar, y_bar, and z_bar spectral matching function data can accuratelyrepresent a theoretical color observer, these data do not representactual people, but rather an average over a group of people (7 peopletested by Guild, and 10 people tested by Wright). A problem with thistesting approach is that there are substantial variations betweenindividuals, and even variations for a single individual over time.There is theoretically an infinite number of different visible spectrawhich can integrate with x_bar, y_bar, and z_bar to yield the same valueof CIE 1931 X, Y, and Z. This transformation of different spectra to thesame CIE XYZ color, and the degree to which a different perceived coloralso sometimes results, is known as “metamerism”. Colors with identicalspectra are called “isomers”, and such colors will always be perceivedby each viewer as identical. However, even though twospectrally-identical colors are perceived as identical to each other, itis not possible to know how the color will appear in the mind's visualfield of various individuals. There is variation from one individual toanother, since each person essentially has his or her own individualspectral sensing functions at any given time. There may also bedifferences between individuals in the neural processing in the pathbetween the retina and the visual cortex, and the resulting perceivedcolor image. Further, there is evidence that people who need tocarefully specify and distinguish colors, such as painters,cinematographers, and interior designers, may learn to improve theirability to distinguish and interpret colors compared to people who havenot been trained in careful color distinction. Moreover, it is not knownthe degree to which color vision is hereditary due to human DNA codes,the degree to which it is neurologically developmental during earlychildhood, and the degree to which it can be further developedsubsequently.

The CIE 1931 y_bar curve, which is based upon photopic luminance, isknown to be incorrect based upon more accurate measurements donesubsequent to 1931. This is significant in the CIE 1931 XYZ color model,since it affects color accuracy and not just luminance perception. Theoriginal photopic luminance work was performed by Judd in the early1920's, and Judd later made a substantial correction in 1950 (see W&SSection 5.7.2), which became the basis of the 1970 Vos version of theXYZ color model (W&S section 5.5.2).

CIE 1964 Supplementary Standard Colorimetric Observer

In 1964, partly due to Judd's correction of y_bar, and also due to thediscovery that a 10-degree color patch matched differently from a2-degree color patch, the CIE added the CIE 1964 10-degree SupplementaryStandard Colorimetric Observer (W&S section 3.3.2). The difference inaverage spectral functions between 2-degrees and 10-degrees is primarilycaused by the yellow tint over the “macula latea” portion of the foveaof the human eye (the macular yellow pigment). This structuraldifference in the eye means that smaller patches of color are moreyellow-filtered than larger patches.

There are thus a number of x_bar, y_bar, z_bar spectral color matchingfunctions representing human color vision, with significant differencesbetween them. FIG. 1 is a graph showing several existing x_bar, y_bar,z_bar color matching functions: the CIE 1931 (2-degree) version, the Vos1970 version, and the CIE 1964 10-degree version.

Currently, CIE 1931 XYZ (and associated x_bar, y_bar, z_bar) isprominent (e.g., in the ICC color standard) as an engine fordevice-independent colorimetry. RGB representations are also the norm incolor work, along with other three-primary (also called “tri-chromatic”)color representations, such as YUV, YPrPb, YCrCb, YIQ, LAB, and LUV. Allof these tri-chromatic representations are fundamentally specified usingCIE 1931 xy chromaticity coordinates for the color primaries, andassociated matrix and gamma transforms, or are specified directly interms of a transformation from CIE 1931 XYZ. Thus, at the present time,every common color system is specified in terms of the CIE 1931 x_bar,y_bar, z_bar spectral mapping.

Human Vision

In addition to color differences between various photopic x_bar, y_bar,z_bar trichromatic color matching functions (shown in FIG. 1), otherfactors affect color sensing, such as low light levels in common colorviewing environments (e.g., motion picture cinema theaters).

The human eye is composed of “rods” and “cones”. Cones operate in highlight levels, and see in color. This is known as “photopic”. Rodsoperate in low light, and see without color. This is known as“scotopic”. Movie projection operates in “mesopic” vision, such that thelower brightnesses are scotopic and higher brightnesses are photopic.The scotopic-luminance center wavelength is broadband centering in cyan,and is thus substantially bluer than photopic luminance which centers inyellow-green. Wyszecki and Stiles discuss “Tetrachromatic” colormatching functions in section 5.6.2 as a means of dealing with mesopicvision. There is also some evidence that a small percentage of womenhave two slightly different wavelengths of red cones (in addition togreen and blue), and can therefore potentially distinguish colors morefinely than people with a single type of red cones. These women are thussomewhat “tetrachromatic” at high brightness, and thus potentially havefive spectral wavelengths active in mesopic vision. It is further commonfor several percent of the male population to be somewhat colordeficient (degrees of color blindness). This usually takes the form oftwo high-brightness cone colors instead of three, but there are otherforms of color deficiency as well. All of these inter-personalvariations make the goal of person-independent color difficult (orperhaps impossible).

It is also worth noting that tiny pigment wafers exist within coneswhich are gradually bleached by light during each day. These pigmentwafers are gradually absorbed at the base of each cone during sleep eachnight, while new pigmented wafers are created at the top of each cone onthe retina. A lack of sleep, especially when combined with thepigment-bleaching effects of high light levels (such as being outdoors),can therefore directly affect color perception. It is also likely that abetter understanding will also develop over time with respect to theseeffects, particularly the affect of sleep and lack of sleep on colorperception. Also, given the nature of this pigment-wafer cycle, it islikely that there may be a difference in color perception in the morningversus the middle of the day versus the evening. This is particularlyinteresting in light of the common practice of seeing movies andtelevision in the evening, sometimes watching until late, just prior tosleep.

CIE170-1:2006 Modified CIE Colorimetric Observer

The CIE 170-1:2006 document, published by the CIE in January of 2006,allows “cone fundamentals” to define color mapping functions as afunction of both age and angle of view to yield a “Modified CIEColorimetric Observer”. For example, FIG. 2 is a graph of variation ofcone fundamentals in CIE 170-1:2006 as a function of viewing angle (1deg, 2 deg, 4 deg, and 10 deg) for age 35 years, and FIG. 3 is a graphof variation of cone fundamentals in CIE 170-1:2006 as a function of age(20 yrs, 40 yrs, 60 yrs, and 80 yrs) for a 2 degree viewing angle. Asshown in FIG. 2 and FIG. 3, the average cone fundamentals (called l forlong, m for medium, and s for short) vary significantly with viewingangle and with age. Further, sensitivity in certain wavelengths changesmuch more than at other wavelengths. Note for example the substantialvariations near 500 nm and near 600 nm. The relative proportions of l,m, and s fundamentals can be seen to show large variations at these andat other sensitive wavelengths.

To quote Section 1.2 of CIE 170-1:2006: “Since observers, even withinthe same age bracket, may differ, a fundamental observer must be atheoretical construct based on averages. Any “real observer” will bedifferent from the “Modified CIE Colorimetric Observer”. The startingfunctions are average colour-matching functions from a large sample(about 50 observers). An important inter-observer source of variabilityis the polymorphism of the photopigments, showing up as small, probablyhereditary, variations of a few nanometers in peak wavelength of thecone fundamentals. Additional parameters such as the lens opticaldensity, or macular pigment optical density, are given as averagefigures.”

The variations become particularly significant in light of the humanvisual ability to perform color matches at substantially less than 1%.It can be seen that the affects of age and viewing angle result in colordetermination variations which are more than an order of magnitudelarger than 1% at sensitive wavelengths. Thus, any system for presentingprecise and accurate color must take into account age and viewing angle,as well as inter-observer variations.

Note also that the CIE 170-1:2006 Modified CIE Color Matching Functionsare based upon the 1959 Stiles and Burch data, as shown in FIG. 3(5.5.6), FIG. 4 (5.5.6), and FIG. 5 (5.5.6) of Wyszecki and Stiles. Thisdata shows variations on the order of several percent (and sometimessignificantly more) between the 49 observers. Similar variation plots(which are clearly approximate) are shown in FIG. 1 (5.5.6) for the CIE1931 observers used by Guild and Wright (7 and 10 observers,respectively) used in 1928-1929.

It should further be recognized that visual science is still activelyinvestigating many relevant issues, and that there will likely besignificant further refinements and improvements in visual spectralsensing models in the future. Further, there is likely to be a gradualimprovement in understanding with respect to inter-personal variationsin color matching functions, as well as factors related to heredity,ethnicity, age, gender, and viewing conditions. There may perhaps evenevolve a better understanding of cultural preferences and biases incolor appearance, as well as factors affecting color distinctionincluding whether accurate color perception is used in one's employment(such as the work of a cinematographer, who is highly trained in thecreation, management, and distinction of color).

The Use of Logarithmic Printing Density from Film Negative

Heretofore, movie masters have been film negative (or film duplicatesthereof), a digital representation of film negative (using printingdensity units), or a reduced-range digital representation such as adigital television distribution master. For digital representations,there may also be many intermediate versions, but none of these isconsidered a master. Usually, only the reduced-range digital versionembodies the final intended appearance, but it does so at the sacrificeof colors and brightnesses which are out of range for the digitaltelevision or digital cinema representation (as determined duringmastering).

In 1990, the present inventor presented a paper at the Society of MotionPicture and Television Engineers (SMPTE) conference (see “The Use ofLogarithmic and Density Units for Pixels” by Gary Demos, SMPTE Journal,October 1991, pp 805-816), proposing the use of logarithmic and densityunits for pixels. Based upon this paper, and the present inventor'srequest to SMPTE for a standardization effort, the SMPTE DPX (DigitalPicture Exchange) file format was created to represent film negativedensity for motion pictures. As part of the SMPTE specification, adensity representation known as “Printing Density” is used, which is thedensity of the film negative as seen by printing film (positive film forprojection). However, printing density is not defined according to anyindustry-wide standard, and is thus difficult to use accurately orprecisely. The more common standardized densities, known as “Status M”for film negatives, and “Status A” for print film, provide moderateaccuracy for use in film developing consistency, but are insufficientfor full spectral characterization.

In addition, negative films vary in their spectral sensitivity functionswith exposure and color. Cross-color terms, such as variations in theamount of red in red, versus the amount of red in white or yellow, arealso significant in negative film. Some of these effects occur due toinherent film light sensitivity spectral functions, and some occur dueto chemical processes and dye-layer interaction during developing, andsome occur due to dyes interacting with light spectra during the makingof print film or scanning in a digital scanner. A further set of issuesarise with the exposure, developing, and projection of motion pictureprint film, which is exposed from film negative. This process becomesmore complex when camera photography uses one film negative, which isdeveloped and scanned by a digital printer, and then manipulateddigitally and recorded to another film negative in a digital filmrecorder (which is then printed to print film for projection). Copy filmnegatives and positives, known as film intermediate elements(internegative and interpositive) are also needed to provide thousandsof film print copies for movie theater distribution.

When projected, a film print's dye spectral density is concatenated withthe spectrum of the light source (usually xenon) and the spectralreflectance of the screen, to produce the spectrum which reaches the eyeof the movie viewers. See FIG. 4, which is a graph of projected filmspectra for red, green, blue, yellow, cyan, magenta, and white. Printfilm uses cyan, magenta, and yellow film dye layers to modulate red,green, and blue, respectively. The spectral transmittance of the variousdensities of dyes do not vary linearly, but rather alter their spectrashapes as a function of amount of density in that and other dye layers.

In addition to all of these film-based issues, the digital informationfrom scanned and processed film and digital camera input may also bedirectly digitally output to a digital projection release master (whichusually also involves compression). Often this master contains asimulation of the film printing process, and its color peculiarities.See, for example, FIG. 5, which is a graph of typical red, green, andblue primaries for digital cinema projectors.

It should be noted that the greatest success in precise color matchinghas come from side-by-side projected or displayed color images, withhuman-controlled adjustment of color to create a match. One suchapproach to the color matching task system is described in the U.S.Patent Application No. 60/198,890 and Ser. No. 09/648,414, “Film andVideo Bi-Directional Color Matching System and Method” in the name ofDavid Ruhoff and the present inventor. In that application, two imageswere placed side by side (or one above the other) on the same CathodeRay Tube (CRT) screen. One image came from a digital film scan (such asusing “printing density” units), the other image came from an electronicor digital camera. Both images were taken of the same scene, and werepresented on the same screen. The colors of the side from the film scanwere given a print simulation, and were adjusted to achieve a match.Therefore, subject to screen uniformity, both images utilized identicalspectra during the matching process. An inverse was created from thefilm image's adjustment and print simulation process, which was thenapplied to the electronic/digital image. The data from the inverseprocess, applied to the electronic/digital image, then matched thedigital film scan. The digital film scan and the inverse processed datawere both output to a digital film recorder, which then receivedidentical RGB input values (in “printing density” units in thisexample). The recorded negative film could then be optically printed andprojected. Since the RGB values from the electronic/digital camerainverse process were very near the RGB values of the original digitalfilm scan as a result of the matching process, the electronic/digitalcamera image would match the film-scanned image of the same scene. Thefilm recorder RGB values were output to a laser film recorder, whichused identical wavelengths of laser red, green, and blue primaries torecord the intermediate film negative. Accordingly, the printing colorprocess which followed was the same whether the original was fromdigitally scanned film or from the inverted electronic/digital camera.Since the spectra of the film recorder was the same in both cases, andthe spectra of film developing and printing were the same (if done atthe same time on the same roll of film), the projected film printresults were identical. Again, the spectra of the digital film scan,when recorded on the film recorder, and the inverted electronic/digitalcamera image, were identical. Thus, nowhere in this color matchingprocess was a color matching function used, nor was it needed. Further,there was no attempt made to match the projected film with its colorgamut, white point, gamma function, and spectrum, to the video image onthe Cathode Ray Tube (CRT), with its color gamut, white point, gammafunction, and spectrum. However, color matching functions, based upon astandardized colorimetric observer, are needed when identical colors aredesired from differing spectra.

In U.S. patent application Ser. No. 11/225,665, “High Quality Wide-RangeMulti-Layer Compression Coding System” by the present inventor,efficient compression coding technology is described which maintains awide dynamic range, and which can preserve the original imageinformation by ensuring that coding error is less than the noise floorof the image itself. The system and method of U.S. patent applicationSer. No. 11/225,665 can efficiently (via compression) preserve anextended color gamut range using negative numbers (for example, in red,green, and blue channels), and numbers above 1.0 (the most commonlogical presentation maximum in a mastering room context) using internalfloating point processing. Additional channels (more than three) canalso be coded. Further, a small amount of bit-exact (lossless)compression is available in the OpenExr 16-bit floating pointrepresentation, although the TIFF-32 standard does not currently offeruseful compression. Without compression, a high resolution movie masteris usually impractically large (many terabytes), even using today'slarge digital storage capacities. U.S. Patent Application No.60/758,490, “Efficient Bit-Exact Lossless Image Coding Residual System”by the present inventor describes how to modestly compress moving imageswhile retaining bit-exact original pixel values.

Color Printing on Paper

Another application where there is a need for accurate and precisecontrol of color is color printing on paper. Color printers for homecomputers are becoming ubiquitous. There is a wide variation in thecolors being produced by various color printers, however. Oneinteresting example concerns some models of color printers which useadditional color primaries (via additional colors of inks). Normal colorprinting modulates red, green, and blue light by using cyan, magenta,and yellow inks (or dyes), respectively. Sometimes black ink is alsoused for efficiency and black quality, although black typically does notcarry any color information. In addition, some printers also use red andgreen as well as low-saturation photo-magenta and photo-cyan inks (ordyes). Some printers also utilize deep blue or violet, as well asorange, deep red, and other colors. In a typical image with red, green,and blue color channels (in addition to yellow, cyan, and magenta),there is little or no information on how to manage amounts of theseadditional colors. Other specialty inks (or dyes) are also sometimesused including “day-glow” fluorescent colors, which often convertinvisible ultra-violet light into visible hot-pink, hot-green, andhot-blue, often with spikes in the resulting spectra. Some printers haveother specialty inks having silver or gold metallic appearance. Somepapers utilize “optical brighteners”, which convert invisibleultra-violet light into increased whiteness, sometimes also withspectral spikes (like day-glow inks and dyes).

The current practice for device-independent color representations fordigital cameras and color computer printers is to use the ICC colorstandard, which is based upon CIE 1931 x_bar, y_bar, z_bar spectralmapping (via CIE XYZ or CIE LAB). This provides only approximatedevice-independence, and provides no information about how to useadditional ink colors (beyond cyan, magenta, and yellow). It should benoted that the mapping between trichromatic images and display orprinting devices with more than three color primaries is non-unique.There are thus numerous possible mappings from three primaries to morethan three. Further, no such predetermined mapping is likely toaccurately and precisely recreate all of the intended colors.

Color printing on film requires the use of a light source forpresentation, with the light source having its own spectralcharacteristics. Color printing on paper requires the paper to beilluminated by light, which light will have its own spectral properties.While the film or color paper print can attempt to reproduce color witha specific spectrum of light, it cannot adapt to other light spectra.

Another significant issue is that the perception of color, brightness,and contrast is greatly affected by the color and brightness of thesurrounding environment when viewing a color print or a displayed orprojected image. The film or color paper print can attempt to correctlyreproduce perceived colors in a single anticipated viewing surround, butcannot be made surround-independent. Further, there is no mechanismwhich could provide for variations between viewers, or within a singleviewer (such as one's gradual adaptation to changing color andbrightness perception when going from bright daylight to a darker indoorroom).

The goal of precise and accurate device-independent color will remainillusive as long as such systems (such as the ICC color standard) areprimarily based upon CIE 1931 x_bar, y_bar, and z_bar. Further, observervariations, and variations in image brightness and in surround color andbrightness conspire to further complicate the task of accurate andprecise reproduction of all of the colors in an image.

Use of CIE 1931 Chromaticity in Specifying File RGB Primaries

In the “High Quality Wide Range Multi-Layer Image Compression CodingSystem” patent application by the present inventor, a compression systemis described which can retain extended range image representations usinga floating point numerical representation. Publicly-available floatingpoint image formats such as TIFF-32 and OpenExr can be directlysupported as inputs and/or outputs for compression or other processing.However, these systems either do not define the spectral mapping ofprimary colors, or else specify color primaries using CIE xychromaticity coordinates (which spectrally map using CIE 1931 x_bar,y_bar, and z_bar). The color precision and accuracy of these formats istherefore limited by the inherent limits of CIE 1931 x_bar, y_bar, andz_bar spectral mappings.

Differences in Spectral Sensing Functions of Cameras with Respect toColor Matching Functions

Another significant aspect of common color imaging systems is thespectral sensitivity function of color electronic cameras (usually theseare digital cameras). Most color cameras use red, green, and bluespectral sensitivities which differ significantly from human colorvision. The red spectral sensitivity of cameras, in particular, usuallypeaks at a much longer wavelength (deeper color of red) than does humanvision. This has the affect of increasing color saturation for somecolors. However, it also has the affect that some of the colors seen bya digital camera will not correspond to colors seen by the human eye(and cannot be unambiguously transformed into colors as seen by thehuman eye).

Camera films (usually negative films in motion picture use) similarlyhave spectral sensing functions which cannot be mapped into humanvision. Camera film spectral sensing functions for the color primarychannels (usually RGB) further vary with film exposure level, withamounts of the other color primary channels, and with otherphoto-chemical factors.

It is currently common practice to alter the original scene colors usinga reference digital display or projector. This process has long beencalled “color timing” for the process of balancing and adjusting colorswithin film prints made from film negatives for stills and movies.

Such alteration can also be regional (such as in photo-retouchprograms), and may be applied as moving image region color alteration(sometimes called “power windows” or “secondary colors” by theterminology of telecine color correction systems). Often these coloradjustments are applied “to taste” and thus bear no relationship to theaccurate nor precise reproduction of colors.

It can easily be recognized that the emission spectrum of the colorreference display or projector is unlikely to match the sensing spectraof digital cameras, which in turn is unlikely to match the way colorsare seen by the human visual system. Further, home viewing and homepaper prints operate using yet different spectra.

There currently exists no practice for usefully reconciling the varioussensing and emission spectra, outside of systems such as the ICC colorstandard (and similar systems) which are fundamentally based on the CIE1931 x_bar, y_bar, and z_bar system, and thus do not take into accountany of the common spectral variations in a typical imaging process (fromcamera to display, projector, color paper print, color transparent film,or video screen).

It is also useful to note that television systems in past decades reliedupon a relatively consistent spectra due to consistent use of CathodeRay Tubes (CRT's), with relatively consistent emission spectra.Reference CRT color monitors were commonly used to set reference colorfor scenes. Since nearly all end-user presentation in past decades wasalso via some form of CRT, the use of CIE 1931 chromaticities to specifythe spectral mapping of colorimetry and primaries was adequate to amodest degree (although many displays differed substantially from colorand gamma-curve calibration specifications). However, at present, mostdisplays are no longer CRT's, and have a wide variation in emissionspectra at mastering and at final presentation. Common current computerdisplays and television displays use LCD's, UHP lamps (metal halide)with projection LCD or DLP modulators, and plasma panel displays, eachhaving significantly different emission spectra. Thus, at present, manysystems rely much more heavily on CIE 1931, within which the red, green,and blue chromaticities are specified, to perform the spectraltransformations necessary for reasonable color representation.

Of greater significance to the present invention is the current practiceof discarding spectral information (both sensing and emission,especially the emission spectra of mastering displays and projectors) asimages are processed and distributed. This is true for digital moviemasters (which are relatively recent), digital television, web-basedcomputer color image presentation, and color photography and color paperprinting for personal and professional use.

In the case of digitally-scanned film negative, it is common practice todiscard the filmstock information, which would identify the spectralsensing functions (although these vary with film emulsion batch), andwhich would identify the dye transmission spectra (which similarlyvary), as well as the inter-layer interaction and exposure versus colorinteractions. Further, the spectral sensing functions of the digitalfilm negative scanner, and perhaps the spectrum of its light source, arealso discarded. What is ubiquitously provided is the RGB data, withunspecified spectral properties. This “raw” color digital negative isthen color “timed” on a reference color projector or display, thespectra of which are similarly discarded.

It is easily seen that accurate and precise control and reproduction ofcolor has heretofore proven elusive. Thus, an on-going technicalchallenge is attempting to accurately and precisely specify color in amanner that is device-independent, person-independent, color-patch-sizeindependent, and time-independent.

SUMMARY

The present invention describes methodologies whereby spectralinformation and/or extended dynamic range is retained as images arecaptured, processed, and presented during color adjustment. Using thisextra spectral information, various methodologies for further presentingor processing the color within these images can be optimized.Presentation-device independence is achieved not by attempting todiscover a device-independent intermediate representation, but rather bydeferring the binding and mapping of color representation onto apresentation device until its actual use. In this way, future means canbe developed, as technology becomes available, of determining essentialinformation needed for accurate and precise presentation of color. Forexample, future image presentation devices may be able to determine theimage surround, and/or develop knowledge of the specific viewer(s), andpossibly their state of adaptation, and/or the size and brightness ofthe presentation, and/or the white-point of the presentation, andnumerous other presentation-time-specific issues affecting precise andaccurate color. Further, existing available color models can be selectedwhen appropriate. For example, a 10-degree CIE 1964 color model can beused for large areas of color, whereas a 2-degree CIE 1931 color modelmight be used for small areas.

A few color perception models have been developed in recent years whichutilize surround, brightness, and white point information for somepresentation environments. Such models can be utilized in conjunctionwith the present invention when appropriate. Color models which embodylow-brightness mesopic color perception can also be applied forappropriate low brightness portions of presentations, such as motionpicture projection. The present invention is likely to aid in thedevelopment of more accurate and precise presentation models, since manysuch models are presently hindered by being limited to tri-chromaticinputs specified in terms of the CIE 1931 2-degree XYZ color model andthe many systems based upon it (including television systems, which havered, green, and blue primaries defined in CIE 1931 xy chromaticitycoordinates).

Some of the more significant new aspects of the present invention forimproving the accuracy and precision of presented color include thefollowing, which may be used alone or in combination:

One aspect of the present invention is to defer the “binding” of colormatching functions and optimal spectral composition (if more than threeprimaries are used, e.g., a white channel) by sending (to and fromstorage, or by transmission) the mastering spectra (three or moreprimaries) along with the corresponding image (whether coded oruncompressed), or by knowing the mastering spectra implicitly. Note thatthis aspect allows the present invention to take advantage of futureadvances in the understanding and modeling of human vision.

Another aspect is to incorporate scene understanding, both of where atypical viewer will be looking (e.g., using eye-tracking data), as wellas the nature of what is being seen (e.g., knowledge of the extent ofregions of color) to adjust for the yellow macular pigment, and toaffect the viewing angle aspect (e.g., 1 deg to 10 deg) of selecting thecolor matching functions.

Another aspect is varying the color matching function with every pixelin a gradual regionally-varying manner, and/or varying the colormatching function as a function of the color of each pixel (with aknowledge of relationship of each color to the color within itsimmediate surround).

Another aspect is to understand inter-observer variations, and that itis feasible and practical to determine and functionally characterizethese variations. These variations can then be incorporated into thepresentation of still and moving images (either to averages of actualviewers, or optimized for one or more specific viewers or families ofviewers).

Another aspect is that it is practical and feasible to take into accountthe viewing surround when presenting images, and to include thisinformation along with the other information generated in accordancewith the above aspects to improve the accuracy of perceived color, aswell as the accuracy and precision of presented color.

The details of one or more implementations of the invention are setforth in the accompanying drawings and the description below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a graph showing several existing x_bar, y_bar, z_bar colormatching functions: the CIE 1931 (2-degree) version, the Vos 1970version, and the CIE 1964 10-degree version.

FIG. 2 is a graph of variation of cone fundamentals in CIE 170-1:2006 asa function of viewing angle (1 deg, 2 deg, 4 deg, and 10 deg) for age 35years.

FIG. 3 is a graph of variation of cone fundamentals in CIE 170-1:2006 asa function of age (20 yrs, 40 yrs, 60 yrs, and 80 yrs) for a 2 degreeviewing angle.

FIG. 4 is a graph of projected film spectra for red, green, blue,yellow, cyan, magenta, and white.

FIG. 5 is a graph of typical red, green, and blue primaries for digitalcinema projectors.

FIG. 6 is a diagram showing positions for black-to-white gray intensityramps around an image, which provide a white, gray, and black referencesimultaneously.

FIG. 7 is a diagram of one embodiment of a scanning spectral radiometer.

FIGS. 8A-C are a diagram showing an overview of the system and method ofthe present invention. FIG. 8A shows an example of a mastering room witha viewing screen on which an image is projected. FIG. 8B shows anexample of a presentation room with a viewing screen on which thepresentation image from the mastering room source is displayed orprojected. FIG. 8C is a diagram showing the types and flow of data foraccurate and precise reproduction of color in accordance with thepresent invention.

FIG. 9 is a graph showing the spectral output of common broad spectrum(white) radiators compared to the theoretical equal energy white (E).

FIG. 10 is a graph showing the spectral characteristics of one selectionof RGB primaries and a broad spectrum white source (e.g., a highpressure 2 kW xenon arc lamp).

FIG. 11 is a graph showing an example of one narrow and one broadprimary for each of red, green, and blue.

FIG. 12 is a graph showing dual R, G, B primaries.

FIGS. 13A-13C are graphs showing cone fundamental ratioslong/(long+medium) and medium/(long+medium) (top), long²/(long+medium)and medium²/(long+medium (middle)), and abs(delta(long))/(long+medium)(bottom), showing variation as a function of age for 20 years, 40 years,60 years, and 80 years for 2 degrees.

FIGS. 14A-14C are graphs showing cone fundamental ratiosmedium/(medium+short) and short/(medium+short) (top), andmedium²/(medium+short) and short²/(medium+short) (middle), andabs(delta(medium))/(medium+short) (bottom), showing variation as afunction of age for 20 years, 40 years, 60 years, and 80 years for 2degrees

FIGS. 15A-15C are graphs showing cone fundamental ratioslong/(long+medium) and medium/(long+medium) (top), andlong²/(long+medium) and medium²/(long+medium) (middle), andabs(delta(long))/(long+medium) (bottom), showing variation as a functionof angle for 1 deg, 2 deg, 4 deg and 10 deg for 35 yrs.

FIGS. 16A-16C are graphs showing cone fundamental ratiosmedium/(medium+short) and short/(medium+short) (top), andmedium²/(medium+short) and short²/(medium+short) (middle), andabs(delta(medium))/(medium+short) (bottom), showing variation as afunction of angle for 1 deg, 2 deg, 4 deg and 10 deg for 35 yrs.

FIG. 17 is a diagram of an image 1700 showing different angular valuesthat may be selected for color matching functions based on variousviewing angles.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION Rich-Content Moving Image Masters

It would be useful to create a system which can:

(1) Preserve the full range of the original image data;

(2) Accurately and precisely specify the intended color as mastered;

(3) Provide all colorimetric and spectral information available aboutthe image, and optionally the intermediate elements which comprise thatimage, to support future improvements in color representation and colorpresentation;

(4) Accurately and precisely specify intended color in alternate usefulpresentation environments (such as home television, digital projectionfor cinema, home and office computer displays, laptop computer displays,color paper prints, etc.);

(5) Communicate the presentation appearance of an image on futuredisplays, future transparent film prints, future paper prints, and/orfuture projectors, when dynamic range and/or color range can be extendedbeyond that available at the time when the image was mastered;

(6) Provide optional compression;

(7) Provide for one or more versions and/or one or more digitalintermediate elements along with the master image;

(8) Adapt and optimize images to one or more viewers; and

(9) Minimize inter-observer variation in perceived color and appearanceof images.

A fundamental premise of the present invention is that relevant spectraland/or dynamic range information about images should not be discarded,even if there is no way at present to use many aspects of suchinformation in display or projection (although potentially useful as asource of re-mastering or re-interpretation, as with film negative). Itis likely that such information will be valuable for display andprojection as color science advances in both theory and practice, and asdisplays and projectors expand their capabilities.

Another fundamental premise of the present invention is thattrichromatic pixels (e.g., rgb or xyz or yuv/yPrPb) specified via theCIE 1931 x_bar, y_bar, z_bar standard are inadequate as a precise oraccurate color representation (although adequate as an approximate colorrepresentation). Current knowledge of color science is still somewhatapproximate. It is therefore beneficial to defer the commitment tomappings into device-independent representations, since current versionsof such device-independent representations (e.g., CIE 1931 x_bar, y_bar,z_bar spectral mappings) are known to be only approximate (and thusprovide only limited accuracy), and are known to apply only to certainuseful cases (e.g., 2-degree color matching for some people in their20's in a dark viewing surrounding).

Additionally, there is known to be substantial variation in colorperception between individuals. The CIE 1931 x_bar, y_bar, z_bartwo-degree and the CIE 1964 x_bar_10, y_bar_10, z_bar_10 10-degreestandards, as well as the CIE 170-1:2006 color matching functions, arebased upon an average over a number of test viewers. Each individualviewer will vary from the average, and the variation is significant formost viewers. Further, color perception changes significantly with age,such that people in their 20's see color quite differently from peoplein their 50's or 80's. The amount of blue light that impinges on theeye's sensing retina is reduced with each decade, due primarily toyellowing of the lens. No single device-independent tri-chromaticrepresentation, such as the CIE 1931 XYZ standard, can take into accountsuch interpersonal variations. The CIE 170-1:2006 standard does accountfor age and viewing angle variation for the average viewer. However, CIE170-1:2006 standard represents a continuous function of age and viewingangle, and thus represents a whole family of color matching functions(unlike CIE 1931 XYZ, which only represents the average for a singleage, being researchers having ages in the 20's, and for a 2-deg viewingangle).

An aspect of the present invention is the understanding that wheneverthe presentation spectrum matches the mastering spectrum, all colorswill be an exact match for all viewers, since identical spectra have nointer-personal variation, and are not affected by variations in colormatching functions. However, today's displays and projectors exhibitwidely varying spectra, having very much greater differences than didCathode Ray Tubes (CRTs) which were the dominant display type fortelevisions and computers a decade ago.

Therefore, this invention retains much more information about spectralmapping (both sensing and emission) through the image processing stepsfrom camera (or synthesis) to presentation, with particular emphasis onthe mastering display or projector.

It is a further premise of this invention that it is best to retainwhatever information is available about spectral mapping of color data,rather than attempting to collapse such data into any current form of“device independent” representation such as the trichromatic CIE 1931XYZ system. Thus, the present invention avoids premature imposition of asingle (and often limited) color representation.

For example, some portions of an image, such as sky or lawn, may occupymore than 10 degrees in the field of view, and may better utilize theCIE 1964 10-deg XYZ standard than the CIE 1931 2-deg XYZ standard (ifthe pixel data is tri-chromatic). Further, the use of the CIE 170-1:2006standard allows the color matching functions to be a continuouslyvariable function of viewing angle between 1 degree and 10 degrees. Asanother example, dark regions of an image may be creatively interpretedusing mesopic vision emphasis or de-emphasis (more or less color thanrod-vision would indicate). Similarly, the interpreted presentation ofan image may be intentionally desaturated (low amounts of colorfulness)and/or warm (reddish or yellowish) or cool (bluish or cyanish) tints,without requiring that the pixel data itself be desaturated, or colorbiased.

As another example, representations, color vision models, orpresentation devices, may be able to utilize spectral mappinginformation to support more than three primaries (e.g., more thantrichromatic). For instance, a commonly available and affordablecomputer paper printer currently uses five primary colors (red and greenare added to the normal yellow, cyan, and magenta as well as neutralblack). Also, a number of common small computer presentation projectorsuse a white/clear segment in a rotating color filter wheel along withred, green, and blue, in order to provide more white brightness. Thewhite segment provides a neutral white primary, in approximately thesame spectrum as the projector's white lamp.

Another fundamental concept of the current invention is that pixelvalues, whether tri-chromatic or more than three primaries, can be usedas indices and weightings into spectral functions (which will alwayshave more than three wavelength weights, and which may have dozens oreven hundreds of wavelength weights). Further, optional additionalinformation, either associated with each pixel, or applied regionally,can select between spectral functions or weight which spectral functionsare to be applied to yield colors in a given situation. Further,optionally there may be more than one such selection and weighting, withrecommendations about when to use them (e.g., such as withlow-dynamic-range displays, or with high ambient light presentation).For example, a color image comprising pixels representing at least aprimary colors can be transformed by mapping the pixels of the a primarycolors to b spectral weights, where b equals at least a+1, and thenintegrating the b spectral weights with c sets of m presentation and nperceptual weights, where c equals no more than b−1 but is at least 3,resulting in c presentation primaries.

Current practice can be seen to be a degenerate case of this concept,without regional attributes (the entire screen being a single uniformregion) wherein tri-chromatic colors are spectrally interpretedaccording to fixed, non-varying spectral weightings of primaries whichare specified in terms of the fixed spectra of the CIE 1931 x_bar,y_bar, and z_bar standard (via xy chromaticity coordinates as thespecification method of the three color primaries, usually red green andblue).

Use of the Word “Primary” in the Present Invention

In general usage, the primary colors of an additive primary color system(e.g., RGB) are visually and mathematically independent of each other'samounts, making it possible to usually just sum the independent amountsof each primary's spectra to determine the resulting summedmulti-primary spectrum. For color paper prints, transparency prints, andother subtractive-color primary systems (most such systems having cyan,magenta, and yellow primaries to modulate red, green, and bluerespectively), all independent color modulating spectra channels arealso generally (and in the context of this invention) called primaries.However, subtractive primaries generally alter each other's spectra andalter each other's amount of color transmission, and are thusinter-dependent. This characteristic usually requires cross-colorcorrections via 3-D (or more than three dimensions if more than threeprimaries are used) cross-color lookup tables and interpolation. Theresults of the cross-color lookup and interpolation might best be entirespectra, although independent weights of a small number of spectra willoften be sufficient.

In the present invention, the word “primary” refers to any independentcolor channel of a digital projector or digital display, and not justred, green, and blue. Such primary colors, as used here, may includewhite, and numerous colors other than red, green, and blue. Further, ifthere are two independent channels for red (or green or blue) with twoindependent spectra (perhaps one narrow and perhaps one wide), thenthese are also considered independent primaries, even though they bothmay be perceived as the same color (e.g., variations of red).

Spectral Characterization of Mastering Displays, Projectors, and theirEnvironment

In the present invention, there is no need to precisely assign aspecific trichromatic (or more than three primary) device independentsystem, such as the CIE 1931 2-deg XYZ standard, at the time ofmastering, since the trichromatic (or more than three primary)device-independent representation can be deferred. For example, a giventransformation might only be applied by the manufacturer of analternative projection or display device at the time of presentation (orsub-distribution to venues having that device). The essential enablingrequirement for specifying a specific color representation for an imageis that the three or more color primaries of the mastering display orprojector be known in terms of their emission spectra. If a particularprimary is set to its maximum value, and all other primaries are set totheir minimum value, the spectrum of such primary can be measured usinga spectral radiometer. Typical spectral radiometers measure to aspectral accuracy of between 1 nm and 5 nm over the visible rangebetween 380 nm and 780 nm and typically have an energy accuracy ofapproximately one percent.

If there is a known transfer function (such as an exponentiation by adefined value, typically called gamma), then the linear amounts of eachspectra from the corresponding linear amounts of each primary can besummed to know the spectrum of a given color on the mastering screen ordisplay.

Of course, the above scenario is overly idealized. The main confound isthe interaction of colors on a screen in a particular room. For example,there may be red seats in a mastering theater, and light scattered backfrom such seats will affect the colors on a screen. As another example,a 2000:1 contrast range may shrink to a 200:1 contrast range in thepresence of large bright areas on the screen, due to light spill comingfrom the projector lens, the projector booth window glass, and fromlight scattered back onto the screen from the room.

Thus, in order to improve the above scenario, a full characterization oflight-bleed interactions would also be useful in the mastering suite.Importantly, having that additional information, it would be possible todetermine, for each frame, via an integration of the light and colorover the entire image, the actual spectral-radiometric color emitted bya region of a given size in the presence of the rest of the scene, andin situ in a theater. In this way, red seats in a theater would becomeindirectly part of the calibration, since their affect on the colors(which the colorist and/or cinematographer may have nulled out) could betaken into account. This step also implicitly measures spectral energyinteractions between primary channels, if present. Such interactions canbe characterized by sweeping all possible combinations of values of theprimaries (using high order bits, and 3-D cross-color lookup, or morethan three channels, and then interpolation). Such methods are requiredfor film due to the existence of substantial inter-color interactions.However, for digital projectors and displays, often the (three or more)color primary channels will be fully independent of each other as theyexit the projector or display screen.

When this color-interaction information is applied to there-transformation of an image for presentation on an alternativeprojector or display device, the same full characterization would beuseful, including the specific room interactions. In this way, thefuture alternative projector or display device, in situ, could reproducethe image as originally seen in the mastering suite, using whatevermetameric transformation best suits that projector or display device.

Further, this concept can be generalized to mastering andalternative-device-presentation using more than three primaries. Thus, aheterogeneous mixture of tri-chromatic and multi-chromatic (fourprimary, six primary, seven primary, etc.) projectors and displays canbe used in both mastering and presentation, taking full advantage of theextra color channels to reduce metameric disparities (i.e., to reduceinterpersonal variations).

Note that a digital cinema master can simply be RGB in its own inputprimaries, with no transformation, and the rest of the inventive systemwould still work. Further, it would not matter if the digital cinemamaster were specified as being under the CIE 1931 XYZ standard (with orwithout a gamma exponent), if the transformation back to originalprojector and/or display primaries (typically RGB primaries iftrichromatic) is known, since the linear RGB values can be reconstructedand utilized as weights for the spectra of each such primary.

In this light, when the mastering projector or display is trichromaticit would be simpler just to master in the RGB input space of thespecific mastering projector or display being used. However, if theparticular display or projector were not calibrated in its specificenvironment (its specific room), this would not work perfectly (due tothe need to characterize room affects on the mastering display orprojector). However, a neutral characterization, typical of a similarquality mastering room with a similar display or projector could beapplied, especially if the specific primary spectra were measured (e.g.,at maximum value for each primary) from the actual projector or display.It is reasonable to anticipate that mastering rooms would be motivatedto fully calibrate their displays and projectors within their actualroom environment, however.

See below for a detailed description for how to optimize the use of morethan three mastering primaries.

Basic Mastering Room Characterization Using RGB

A basic characterization of any mastering room, whether projection ordisplay, can be obtained by determining the spectra for full amplitudeof each primary (usually R, G, and B).

In addition, it is important to understand the mapping between digitalpixel values of the image and the light that is emitted from aprojection screen or an image display. It is often possible to ignoremost inter-pixel effects, and determine an effectivepixel-value-to-light mapping, which is often intended to be close to a“gamma” exponent, with a gamma of approximately 2.2 being common(although the entire range between 1.6 to 2.6 is utilized for variousdisplays and projectors).

If the primaries vary differently in their pixel value to lightfunction, then each primary must be measured or approximated (e.g., byits approximate gamma) independently. If the function is not gamma(i.e., not an exponent), then a lookup table (possibly created orimplemented with interpolation) can be used to yield the appropriatepixel value to light function. Gamma, log, quasi-log, linear, and othermappings from pixel values to light are also useful and may beimplemented via lookup tables (possibly with interpolation) or by usingthe appropriate functions.

Note that some types of display and projection devices do not exhibitsignificant inter-primary interactions, whereas others can havesignificant interactions. When such interactions are present, a morecomplex characterization is required which can appropriately account forthese interactions (see discussion below).

Note that current practice is to assume a pure gamma function, and acomplete absence of inter-primary affects. It is also current practiceto ignore the spectra of the primaries, and only to utilize CIE 1931 xychromaticities during calibration when calibration is possible, and toignore any actual variation from color primary xy chromaticitycalibration otherwise.

One key concept of the present invention is that the determinedmastering room display or projector primary color spectra be retainedand conveyed with each group of images mastered with those primaryspectra. An additional key concept is that such spectra be utilized forpresentation (see discussion below).

This differs from current practice, wherein the mastered RGB values arereferenced to their CIE 1931 xy chromaticities (whether the masteringdisplay or projector was properly calibrated or not). Any subsequentcolor transformations, such as RGB to YUV (or to YPrPb, to YCrCb, toXYZ, etc.) are performed as matrix operations (either 3×3 or 3×4matrices), without further reference to mastering spectra. Note that thematrix multiplies in current practice are often applied to non-linear(e.g., gamma) pixel values (which is common practice in television, butis an incorrect practice). The SMPTE DC-28 specification for matrixprocessing of pixels for digital cinema projection utilizes CIE 1931 XYZwith a 2.6 gamma applied, which is removed (yielding correct linearpixel values) prior to applying the 3×3 matrix transformation to RGB forprojection. This still is dependent upon having collapsed the RGBmastering spectra into the CIE 1931 XYZ representation.

Note that calibration chromaticities may not be achievable for everysituation. In such cases, current practice is to get as close aspossible, and then to ignore the remaining difference in chromaticity.Even this could be improved by conveying the actual CIE 1931 xychromaticities which were achieved during calibration, rather thanignoring the difference. However, a better practice is to send theresulting primary spectra (as is proposed by the present invention),since this eliminates any dependency upon CIE 1931 xy chromaticity, anddoes not depend upon whether the projector or display can achieve thecalibration chromaticities.

Note also that this aspect of the present invention (conveying masteringroom display or projector primary color spectra with each group ofimages mastered with those primary spectra) can be implemented entirelyas matrices, if the integrals with color matching functions and spectraduring mastering and presentation are known a priori, as well aspossibly other viewing and mastering conditions. A set of matrices,corresponding to the known conditions, can be used. When adjustments forvarious attributes (such as viewing angle, see discussion below) aremade, these adjustments can be implemented via interpolation (andpossibly renormalization) between the matrices within the set.

Thus, although the present invention requires the use of the integrationof spectra for mastering and presentation, such integration cansometimes be performed a priori, resulting in an implementation entirelyusing matrices. This differs from current practice, which uses a singlematrix for any given display, where that matrix is based solely on CIE1931 x_bar, y_bar, z_bar integrations (computed via either xychromaticities resulting from these integrations, or XYZ tristimulusvalues resulting from these integrations).

More than Three Primaries

Use of more than three primaries in the mastering room is a simplelinear extension of the above method. For example, four primaries ormore can also have their spectra measured by giving a maximum valuesolely to each color channel.

Note that there is no current practice with respect to more than threeprimaries, nor for primaries other than red, green, and blue.

Idealized Concept Approach for Mastering Room Characterization

A known concept can be utilized in conjunction with the presentinvention, which is to set an idealized black at “zero” light, eventhough total lack of light almost never exists (e.g., the blackest blackof a projector still emits some light, and room light spill on a displayor projection screen lightens “black” areas). Thus, a zero pixel valueis intended to make perfect black, even if it does not actually do so onthe mastering screen.

Another known concept can be utilized in conjunction with the presentinvention, which is to define an idealized uniform screen, such that aparticular RGB value is intended to produce the same color, spectrum,and brightness everywhere on the screen, even if it does not actually doso on a real mastering screen.

Addition of Light Level at Black

The idealized zero-light black concept can be improved if the lightactually emitted from a screen can be measured spectrally for thosepixels that are supposed to be true black. Further, this “blackspectrum”, if measured, should be conveyed along with the measuredprimary spectra (e.g., R, G, and B or more than three primary spectra).Note that light level at black can be measured with common sensitivelight meters, although the chromaticity and/or spectra of black mayrequire long measurements or specialized high-sensitivity chroma metersor spectral radiometers.

As an alternative, a black spectrum can be deduced from low light level(e.g., 1% and 2% of white) spectral differences of three or moreprimaries and gray (gray being all primaries at the same value) if theavailable spectral radiometer is not sufficiently sensitive. This isdone by measuring at the lowest possible light level which is practicalfor the spectral radiometer, and then a light level above that lowestlevel (e.g., one stop higher, which is a factor of two brighter). Theamount that the spectra is altered for the primaries and gray valuesallows the inference by extrapolation to the amount of light, and itsspectra, when the pixel value goes to zero (i.e., black). For example,if a given pixel value is intended to yield half the light (in linearunits) but in fact yields ⅔ as much light, then the ⅙ difference(between ½ and ⅔) is the inferred amount of light from projection screenblack or display black. Similarly, this concept can be extrapolated tothe spectra, by noting, for each wavelength, the amount of light abovehalf, to yield the spectrum of projection screen black or display blackvia inference. For example, if the energy at 470 nm falls to ¾ (insteadof ½). then ¼ of the energy at 470 nm is coming from the display atblack or projection screen at black. At 580 nm, if the energy falls to ⅝(instead of ½), then ⅛ of the energy at 580 nm is coming from thedisplay at black or projection screen at black. If this methodology isapplied to all wavelengths, the spectral energy at black can bedetermined for projection screens or displays.

Under the normal condition where the black spectral energy is anegligible fraction of the full-on-primary spectra, the spectra emittedfrom the screen for any pixel brightness can be determined by summingthe black spectral energy with the linear amount of energy from thelinear weight (thus without gamma) of each of red, green, and blue (andmore primaries, if more than three). To the degree that the values aretruly linear, this will accurately yield the spectra emitted from thescreen for each possible value of RGB (or more than three) primaries.

Numerous Non-Idealized Corrections are Also Feasible

If the pixel values in an image are not truly linear, and/or if thereare inter-pixel interactions, then additional corrections must beapplied to better achieve display accurate colors. The model used forsuch corrections must match the nature of the issue being corrected. Forexample, light spill from one area of the screen to another must modelthe amount of light emitted by each area of the screen. As anotherexample, if a screen dims under the load of a large amount of light,then the amount of total light emitted from the screen must beconsidered in order to make a correction. As another example, if thespectra or color balance alters when viewing the screen from differentangles and different locations, the correction must take into accountthe viewer's position with respect to the screen. For example, in somecases, a high-gain non-uniform screen (e.g., a silver screen used for3-D stereoscopic polarization preservation) should not be modeled withrespect to mastering, since the “hot spot” near the image center isentirely an artifact of the screen's high gain, and does not representthe intended image. However, such a hot spot artifact could be correctedduring mastering presentation, and/or during final display or projectionpresentation. A uniform field of each primary color and of white can beutilized, as well as the viewing location, to determine a correction.Such a correction can even account for spectral alteration if theprimaries and white are measured spectrally from a particular locationfor each region of the screen. In such a case, the spectra would need tobe known, and possibly conveyed, for each region of the screen for eachprimary and for white and for black, for each useful viewing location.If the spectra alter in broad spectral balances, a few spectral weightscan be applied across the spectra. For example, a smoothly varyingspectral weight might simply have a red, green, and blue adjustmentamount which smoothly interpolates across the visible (e.g., 380 nm to780 nm) spectra, to weight the primary spectra with respect to viewinglocation or screen position. If the spectral alteration is complex, thenmeasurement and conveyance of the entire spectrum for each imagelocation and/or viewing location may be required.

Elaborate characterizations, as described above, become feasible underthe spectral-retention and conveyance concepts of the present invention.

How to Apply Mastering Room Characterization Information for a GivenPresentation

As noted above, under the present invention, spectral characteristics ofan image display (directly or via projection) in a mastering room aredetermined and conveyed with each group of images mastered in such room.Presentation use of such conveyed characterization information involvesa similar procedure, described below, whether the screen is a display(like a computer monitor) or a projector and screen. Any or all of theabove mastering room characterizations can be applied to a presentationroom and display or projector plus screen.

The first step will usually be to correct image pixels for gamma andbrightness issues.

The next step is to determine the presentation spectra of thepresentation display's or projector's primaries using the maximum ofeach primary (with all others set at minimum).

Conceptually the mastering black spectrum is then optionally summed withthe mastering primary spectra, weighted by their linear-representationRGB (and other primary) values to yield the spectrum of each pixel atthe time of mastering (assuming no idealized black). Then thepresentation system spectrum of black is subtracted (since it is alreadypresent in the presentation environment), and the resulting spectra areintegrated with the appropriate (which could vary per pixel) colormatching functions (e.g., 1 ms cone fundamentals from CIE 170-1:2006) toyield the amount of RGB, or more than three primaries, to utilize forpresentation (see below for a description of the use of more than threeprimaries in mastering and/or presentation). Thus, this aspect of theinvention can be summarized as (1) determining the color spectra and theblack spectrum (and optionally the white spectrum) of the mastering roomimage display characteristics, (2) conveying the determined spectraalong with the mastered image to a presentation system, (3) determiningthe black spectra of the presentation system, (4) correcting themastered image as function of the conveyed determined spectra to yieldthe spectrum of each pixel at the time of mastering, (5) subtracting outthe black spectra of the presentation system from the corrected masteredimage, and (6) mapping the resulting corrected master image to a displaysystem color space to utilize for presentation.

Using 1 nm wavelength resolution intervals over the range 380 nm to 780nm, only 401 multiplies and 401 adds are needed for a given integrationfor each primary. With today's personal computers and display graphicscards, as well as with today's digital televisions, such an integrationcan be performed thousands or more times per frame to provide regional,viewing angle, regional angular size of various colors, and personalvariations. With high end personal computers, high end graphics cards,and high end high definition televisions, it is even feasible to performsuch integrations independently for each pixel. However, since emissionspectra and color matching functions, and their use, should varysmoothly over regions of the image and should continuously vary smoothlyas a function of color, it is usually not necessary to perform theintegration for each pixel. Instead, deltas to color matrices forvarious parameters can be determined once per frame (or per scene orgroup of frames). A weighted sum of such deltas can be utilized whenthey are linearly independent. If they are interdependent, then aweighted sum of the resulting matrices (possibly with renormalization),with matrices at several non-linear key parameter values, can be used tomodel and process all of the attributes described in the presentinvention.

It is perfectly feasible to compute everything needed to buildfunctionally variable matrices at the start of any new show (imagesequence), and even practical at the start of each scene or even eachframe.

Conceptually, the use of weighted matrices or deltas to matrices is acomputational expedient. If one conceives of a single intended spectrumfor each pixel, an effective viewing width angle associated with thecolor of that pixel (determined by the regional size of the given color,and the relationship of that color to the viewing center), and the ageof a viewer (or average age of a group of viewers), a color matchingfunction can be selected to integrate with that spectrum according toCIE 170-1:2006, or any other appropriate color vision model (and itsassociated functionally variable color matching functions). The conceptsin this paragraph are described in detail below. The presentationprimary spectra can then be integrated with the same color matchingfunctions and matrix inverted to yield the matrix to apply to the pixelcolor for presentation (with the addition of gamma, or other functionwhich converts pixel values to linear light).

Note that the integrations and matrix operations described here shouldutilize linear pixel energy (or equivalently linear pixel photon quanta,although energy is common practice and is generally more convenient),primarily because matrix operators are linear operators and are intendedfor use with pixel values which are expressed in terms of linear light(i.e., proportional to foot-lamberts or cd/m² of emitted energy).

If more than three primaries are used in mastering and/or presentation,it is optimal to maximize the broad spectral emitters (see discussionbelow). Conceptually this can also be done on a per-pixel process, butcan use the same computational expedient of deltas or weighted matrixsums (with possible renormalization) since the function should besmoothly varying by color and region, as with trichromatic (usually RGB)presentation and/or mastering.

One possible practical implementation method might be to send some orall of mastering room characterization information to a digitalprocessing system which is driving the presentation display orprojector. The digital processing system can then use this data whenpreparing the pixels for display or projection.

Note that these are just some of the ways to use the mastering roomspectral characterization. There are additional issues, related to sizeof the screen in the field of view at the time of mastering versus atthe time of presentation, the absolute brightness and color gamut ofpresentation, where the viewer is looking (and whether the eye is likelyto wander), viewer age or average viewer age, inter-personal-variationissues, etc.

With current processors and graphics systems, it is perfectly practicalto apply varying matrix values to every pixel depending on the locationof the pixel, the color of the pixel, the location of the viewer andsize of the screen, and where the viewer is looking (and whether theireye is likely to wander the particular scene), plus specific viewer oraverage viewer information.

A color and/or brightness compensation can be computed for differentbrightness levels at the time of presentation versus when mastered. Acolor and/or brightness compensation also can be computed for adifferent ambient surrounding environment (e.g., amount of ambient lightspill, color of the room, etc.) at time of presentation versus whenmastered.

Application of Color Matching Functions to Mastering and Presentation

There are two points at which color matching functions are applied incases where the spectrum of the reference projector/display does notmatch the presentation projector display. The first point is the mappingof the spectra used for reference mastering to one or more colormatching functions. Ideally, the reference viewer(s), which is typicallythe colorist, the cinematographer, and/or the director, arecharacterized at least by age, and possibly also (preferably) by aspecific color matching function. If the reference projector and/ordisplay has more than three primaries, such as a broad-spectrum white,it will be optimal to utilize as much of the broad-spectrum primaries aspossible in order to minimize inter-observer variation. Also, theangular size of the image when performing the mastering is valuable,such that wide areas of color can use 10-degree matching functions forthe reference viewer, small areas of color can use 1-deg matchingfunctions, and intermediate areas can use an “in-between” angularmatching function. The master can be trichromatic, preferably in thespectra of the mastering primaries, or it can have more than threeprimaries, again in the mastering spectra of each primary. For example,for a broad-spectrum white channel, in addition to red, green, and blue,the amounts of the white channel and red, green, and blue channels canall be conveyed as four color primary channels. Alternatively, a knownfunction to convert mastered RGB values into the four color channels(including the broad spectrum white) can be utilized so that the fourchannels can be reconstructed unambiguously from the RGB three channelsbeing conveyed.

The second point of application of color matching functions is when themastered color, whether trichromatic or more than three primaries, ispresented to a viewer for final presentation on a monitor or projector.Using color matching functions aimed at the viewer, or at an average ofa population of viewers (such as a movie audience), and taking intoaccount the angular width for the viewer or the average of viewers, acolor matching function can be chosen for presentation. If a fourthbroad-spectrum white is also included, maximizing its use will minimizeinter-observer variation. Further, to the degree that more than threespectra can be adjusted to match the mastering spectra, this willminimize the difference from the reference viewing (adjusted for age andviewing angle). Further, when mixing more than three primaries, it isuseful to minimize the use of primaries which place increased energy(such as spectral spikes) at points of maximum inter-observer variation,such as deep red (>690 nm), yellow (near 590 nm), cyan (near 500 nm),and blue-violet (<440 nm).

Example Full Mastering Room Characterization

One possible methodology to achieve a full mastering room spectralcharacterization is described as follows:

(1) Select and maintain a consistent calibration methodology for thedisplay or projector which is being used as the reference display in themastering room. It does not matter what calibration methodology is used,as long as the image is repeatable.

(2) Select and maintain an unchanging input representation and anunchanging reference setting (associated with any presentationtransformations normally used) within the mastering display orprojector. For trichromatic (usually RGB) displays and projectors, anytri-chromatic input representation (RGB, XYZ, YUV, etc.) should besatisfactory, but it is likely that RGB will most naturally fit thedisplay or projector, since most such devices emit red, green, and blueprimaries. If more than three primaries are used, then each primaryshould be characterized.

(3) Measure the mapping between all possible primary input values, usinghigh order bits (with the number of bits being selected based upon thenumber of practical measurements). This measurement should use a smallfield, such as a 2-degree square or circle, with a black surround. Themeasurement should be made using the best spectral radiometricwavelength resolution (usually between 1 nm and 5 nm) over thewavelength range between 380 nm to 780 nm. This set of measurementsbecomes the primary characterization of the mastering room, providingthe mapping of tri-chromatic (or more than three primary) projectorpixel input values to displayed spectra.

(4) Measure the light spill onto the small field, using a series ofsmall regions (such as 2-degree squares or circles) at various positionson the screen. (for example, the upper left, upper middle, upper right,middle left, middle right, lower left, lower middle, and lower right).Using these positions, one at a time, measure the affect on a neutralgray patch (such as 18% of peak white), and on a near-black patch (suchas 1% of peak white) of every possible tri-chromatic color (againsweeping high order bits). Each measurement should be made using aspectral radiometer, as in step (3) above. The neutral gray andnear-black patches should also be measured with a black surround. A trueblack measurement would also be beneficial, if the spectral radiometerhas sufficient sensitivity. The black measurement measures not only theminimum emitted light from the display or projector, but also theambient spill light in the room (such as from an exit sign onto ascreen). Note that this black-surround measurement of the gray patchwould establish the reference projection/display white point, and itsentire spectrum. Note also that this white point may be insufficientlycharacterized by a color temperature, and may be insufficientlycharacterized by a CIE 1931 2-deg xy chromaticity value, since the fulldefinition of this white-point (using the 18% gray patch) is onlyprovided by the spectra when it is measured in this step. Iftrichromatic, the white point will be the maximum values, usually RGB,or the 18% gray values. See below for a description of how to optimizethe use of more than three primaries with respect to all colors, but thewhite point in particular. For this step, when there are more than threeprimaries, the 18% gray patch will not have a unique set of values, andwill further be dependent upon the choice of color matching functions.In this case, it is best to measure the 18% gray patch using severalcolor matching functions. Then each primary should be maximized andminimized as well as one or more values in between, while maintainingthe intended neutral gray under each color matching function. Note thatthere will usually be an infinite number of combinations of amounts ofmore than three primaries which will yield the 18% gray patch, and thateach color matching function will require a different balance of theamounts of these (more than three) primaries in order to yield 10% gray(for a given “color” of 18% gray, such as tungsten at 5500K). However,in practice, according to the principles of this invention, gray willgenerally utilize a maximum amount of one or more of the broadbandspectral primaries, and such configuration may be sufficient (althoughit is probably best to still use a number of color matching functions).Note that the maximization of broad-spectrum primaries, especiallybroad-spectrum whites and grays, minimizes inter-observer variation, aswell as variations with respect to differences in color matchingfunctions. As will be described below, a primary corresponding to theequal-energy illuminant “E” will show no variation with respect to colormatching functions (since they are all normalized to equal areaaccording to the theoretical “E” illuminant).

(5) Repeat step 4 with the gray patch and near-black patch moved to eachof the screen edge locations, and adding the middle of the screen to theset of locations for the spill-emitting regions. The set of measurementsfrom steps (4) and (5) become the secondary characterization of themastering room, providing important information about the spectralalteration due to spill from the surrounding image area, as well asinter-color effects that may be present within the electronic projectoror display (if any).

(6) Illuminate a single pixel in the center of the screen, sequentiallyin gray, yellow, cyan, magenta, red, green, and blue, as well as in thecolor of any other primaries which might be present, and all possiblemaximum value combinations of primaries. Using a precisionwide-dynamic-range camera (such as the Thomson Viper™ or an appropriatedigital still camera), align the pixel raster from the projector with aknown number of pixels in the camera. The camera preferably has aspecified high-precision mapping to linear light. The lens used on thecamera preferably should have low near-field and wide-field flare, andpreferably the lens used (in the actual camera, since other opticalelements in the camera may also have an affect) is itself spectrallycharacterized, so that camera optical near-field spill can be nulled outof the measurement. Capture one frame for each primary and combinedprimary color at the center location. Repeat the procedure for the upperleft, upper middle, upper right, left middle, right middle, lower left,lower middle, and lower right. Using this data, the near-field spill foreach pixel can be modeled.

Using the above methodology, a characterization is achieved for thespectral emission from a given pixel value. Further, a characterizationis achieved for the alterations to that spectra due to all sources ofspill in that mastering room. Using the projector's or display's inputpixel values of each actual image subsequently mastered in that room, itis then possible to compute the actual screen emission spectra for everyregion of the screen. This emission spectra can then be used to provideoptimal matching information in support of device independence. Eachalternate device would then have the target spectra for each color, andcould apply whichever transformation (often involving a set of colormatching functions) optimally matches the re-creation of a color havingthat spectral emission.

This display and projector characterization should allow exact or nearexact matching of every color in every reference frame if aspectra-reproduction projector or display (although none arecommercially available at present), which would then exactly reproducethe image. Also, an increase in the number of emission primaries abovethree (i.e., more than tri-chromatic) would be supported in astraightforward way using this characterization and the methodologiesdescribed below. A presentation device population having a heterogeneousmixture of numbers of emission primaries would be also naturallysupported by these methodologies. In this way, there would be a naturalpath to new display technologies, supporting any pace whereby thereevolve increasing numbers of primary colors beyond the initial three(the three typically being RGB).

These mastering room characterization procedures can be further improvedby re-measuring the room regularly, perhaps using an automated orsemi-automated measuring system having a computer-controlledspectral-radiometer. It is feasible to continuously measure the displayor projection system by measuring the colors being presented versus thepixel values being used to make those colors. If a spectral radiometerwhere used for every pixel in a camera (e.g., a spectral-radiometermoving image camera, although none are commercially available atpresent), one could continuously measure the emitted spectrum for theentire displayed or projected image by examining the spectra of thepixels seen by such a spectral camera.

This above methodology assumes pixel-independent behavior in theprojector or display. If an inter-pixel-dependent behavior is present,such as a reduction in the brightness of a given pixel value under theload of making an entire screen bright, then a more completecharacterization will be required to model such conditions. Note thatthis problem occurs intentionally in some plasma displays to lengthenuseful life (by darkening the screen when there is a high percentage ofwhite or bright colors in the image), and that this problem occurs inCRT displays when there is insufficient power supply regulation forstability (again, darkening the screen under the load current requiredto display large areas of white or bright colors).

Regional Variation on the Screen

Note that the mapping of digital values to light energy, typicallycalled video gamma or transfer characteristic, varies with location onthe screen and direction of viewing. This is exhibited as a change inapparent contrast and/or colorfulness at different regions of thescreen, and further changes if one moves one's head (or changes seat ina viewing room).

Note that CRT's exhibited some directional color and brightness andgamma alteration behavior, as do plasma and other displays.Retro-reflective large-screen televisions, which represent a largeportion of existing televisions, have many directional andscreen-location variations that would benefit from these techniques.

Regional Variations Affecting 3-D Stereoscopic Viewing

For 3-D stereoscopic viewing, it is often the case that brightness andcolor differ between each eye, and with viewing location within theviewing room. Color and brightness maps of the screen (either displayedor projected) for any given viewing location can be created for eacheye, either spectrally or via a simplified chromaticity and brightnessmap. Such maps can then be utilized to correct for color and brightnessfor each eye for all regions on the corresponding portion (i.e., left orright eye) of the 3-D display from that viewing location. For multipleviewers, an average map, either uniform or favoring one or more viewers,can be utilized. With multiple viewing locations, such as multipleseats, or even positions within seats for small scenes, it is necessaryto measure or approximate the differences in spectra and/or color andbrightness for each such seat, over all regions of the screen (eitherdisplayed or projected). Accurate color and brightness can then beprovided to at least one seat, and perhaps to many seats near each other(e.g., the center of a theater, or the center of a living room). Againthe knowledge of viewing center (when available) can be utilized withvarious angular-determined (e.g., 1 deg to 10 deg CIE 170-1:2006) colormatching functions, which vary smoothly regionally (and optionally alsoover time) as applied to 3-D stereoscopic viewing.

Note that some 3-D stereoscopic displays require glasses, eitherpolarized or density modulated back-and-forth between the two eyes. Theproperties of the glasses may be affected by where the head is oriented,and thus which part of the glass is being looked through at a given partof the screen. For this purpose, a characterization of the specificglasses, their affect on color and brightness, and optionally spectra aswell, plus head orientation, can be added to the information needed tomake a thorough correction of color and brightness.

For 3-D displays not requiring glasses, such assmall-directional-filters (such as vertical “lenticular” bar screens),the location viewed-from and looked-at may be a sufficientcharacterization of brightness and color, optionally via spectra.However, some such micro-directional filters are so sensitive to viewingposition that it may not be possible to precisely characterize andcorrect such a display. However, an approximate characterization isfeasible for nearly all types of display technologies including alltypes of 3-D display technologies.

3-D systems using polarizers or shuttered glasses (often liquid crystal,which also involves polarization) generally provide the full spectrum ofthe screen or display. However, in recent years, a spectral band-splitmeans of separating the left and right eye images has sometimes beenused. This is done with a deep (longer wavelength) red green and bluefor one eye, and a shallow (shorter wavelength) red green and blue forthe other eye. Such spectral band-split systems rely more heavily oncolor matching functions, since the color seen by each eye will differif correct color matching functions are not applied independently toeach eye based upon spectra received for red, green, and blue primariesby each eye. The methods of the present invention allow more thorough,accurate, and precise application of color matching functions toperceived color, given the necessary spectra. Note that such methods aremuch more sensitive to inter-personal and color matching functionvariation. When colors on an object do not match between eyes, theartifact is sometimes perceived as “iridescence,” as is common onbutterfly wings and when seeing thin film coatings on glass or plastic(or even thin oil film on water). This conveys a very differentimpression than an object having the same perceived color in each eye.

Intentional Idealization

It will sometimes be the case that idealization is intentional. Forexample, it may be the intent that the black pixels produce zero light,or at least as dark as any display or projector can produce, even thougha given projector or display will emit light from the screen when blackpixels are specified. Also, it may be the intent that the screen beuniform, even though it is not actually uniform, such that a given valueof pixel is intended to produce the same amount of light everywhere onthe screen, independent of all other factors. It may also be the intentthat a given object (e.g., a flower) be as colorful as possible, evenwhen the mastering display or projector has a limited range of color.

Such intentions can be optionally specified, and should be sufficientlyspecific that the meaning is as clear as possible with respect topresentation. For example, if a zero pixel value in all primaries meansblack, but a given presentation screen produces 0.1% of white for itsdarkest black, what should be done with a pixel representing 0.1% ofwhite? Several choices might be that a) it remains at 0.1%, or b) itbecomes 0.2% (in effect adding it to the “base” black value of thepresentation screen), or c) it becomes an in-between value at 0.15%(which is probably the best choice). At some point above this“interpreted” presentation black, the pixel value should be smoothlyadjusted such that the pixel is again precisely tracking, such as above0.4% (i.e., so that values above 0.4% of screen white emit light basedprecisely upon the image pixel values, but below 0.4% follow thespecified rules for presentation of black, with such rules taking intoaccount various anticipated black values of various anticipatedpresentation displays). To the degree that such intentions can be givenspecific rules for how to implement them in various situations, suchintentions can indicate precise implementation rules, and thereforeyield specific intentional results in various presentation situations.

Note that current practice in moving images is to provide no“intentions” nor rules for implementing any intentions. For stillimages, the ICC color standard provides for intentions such as“colorimetric accuracy”, “idealized black”, or “best presentation”, butwithout providing any specific rules for how to implement theseintentions. Such implementations are left to each manufacturer of colorprinters to implement, and nearly all such implementations areundocumented. It is often the case that all such vaguely specifiedintentions within “ICC profiles” yield unintentional and/orunpredictable results.

To avoid this problem, intentions should be implemented according tospecific rules. Such rules should either be conveyed explicitly, orconveyed by using pre-arranged and pre-specified rules and identifyingwhich known rules to utilize (possibly according to presentation deviceand/or viewing conditions).

Note that several key people may have different specific intentionsduring mastering. For example, the director, the cinematographer, andthe colorist may all have differing specific intentions for thepresentation of black. In such a case, all of the different intentionsmay be conveyed, with an identification of each “intender” beingassociated with each intention. Note also that a key person may changetheir intention after a time, such as a director or cinematographerchanging their colors or scene gamma when re-mastering for video as aminor re-interpretation. The various intentions, and their associatedtime and context, can each be conveyed. When presenting the masteredimages (either still or moving) on a projector or display, a choice canbe made between the various intentions for a given feature (such as thetreatment of black), if more than one intention is provided for thatfeature. Such a selection can be made by the preference of the viewer orgroup of viewers, or by selection of a specific key person's intent (forexample, choosing the cinematographer's intent versus director'sintent).

Other Uses of Intent

Intent may have numerous other uses besides forms of idealization. Forexample, the location, viewing angle, and screen brightness may be knownfor a key person during mastering, such as the cinematographer ordirector. However, it may be their intent that this screen brightnessand viewing angle are merely a convenient expedient, and may notrepresent their intended image angular size and/or image brightness. Itcan therefore be useful for key mastering personnel to indicate theirintent with respect to their mastering viewing conditions, if suchviewing conditions are conveyed. For example, mastering viewing mightsubtend a 20 degree horizontal angle, but the intent might be for idealviewing to subtend a 40 degree horizontal angle for the width of thescreen.

Further, there may be more than one intentional alteration of a masteredimage which is acceptable to each of the key personnel. For example, acinematographer and/or directory may indicate the specific treatment ofblack and dark gray when being viewed in high ambient surround lighting,and a different specific treatment of black and dark gray when beingviewed in a low ambient surround. A cinematographer and/or director mayindicate the specific color saturation intended for those who wish toview a given show with more or less colorfulness, such that the lowsaturation colors are specified accurately and precisely for thosechoosing low saturation viewing, and that alternatively the highsaturation colors are specified accurately and precisely for thosechoosing high saturation viewing.

Another approach would be that a show is intended to be viewed in a darksurround, with a specified intent for the treatment of black. However,if the show is seen in a high ambient surround, the specifics of blackand dark gray processing are provided by one or more of the keypersonnel. Thus, the intended viewing environment is conveyed, but alsothe presentation is intentionally and accurately and precisely specifiedeven when the viewing environment does not correspond to the preferredintent. A mastering environment may be intentionally altered, such as byadding substantial ambient room light, including shining some light onthe projection screen or display screen, when setting the black and darkrendering specific parameters for such altered high-ambient viewingintent.

When feasible, intent can also be conveyed with respect to alterationsin presentation brightness, colorfulness, ambient surround brightness,white point, presentation contrast, maximum black-to-white ratio,viewing angle, off-angle viewing, and numerous other common alterationsin the presentation environment with respect to the masteringenvironment.

There are likely to be many useful ways in which key personnel duringmastering can convey their intent with respect to one or more alternateviewing conditions and/or viewing preferences.

In the absence of such preference information from key personnel,various common adaptation transformations can be applied. For example,it is common to use a “von-Kries” transformation when altering thepresentation white point with respect to the mastering white point. Muchof the function of “color appearance modeling” deals with attempting toautomate many alterations of common presentation with respect to theidealized mastering presentation. Correcting for alterations in thewhite point is one of the most significant aims of color appearancemodels.

Accordingly, in one aspect of the invention, rules specifying the intentof various content creators are clearly and unambiguously conveyed witheach image being mastered for use in a presentation system to alter thepresentation display of such content in a pre-defined way, according topresentation context and/or viewer choice.

Correction for Linear Light

It is beneficial to specify correction for linear light to cause pixelvalues to have a defined mapping to linear light, or to send adescription so pixel values can be corrected to linear light at thepresentation display or projector. Current common practice is to use animplicit gamma exponent, or a linear-black-adjusted gamma exponent(called a “transfer characteristic” in video system specifications).Alternatively, the projection system can be calibrated to make a knownfunction (or linear representation) of the pixel values into linearlight. As a hybrid, it is useful to calibrate as much as possible, andconvey the remaining information where calibration could not accomplishthe full extent of the required correction(s). The goal is to provide adefined mapping from pixel values into the linear light that was used atthe mastering display or projection screen, so that the same amount oflight may be accurately reproduced during subsequent display orprojection.

Alternatively, if adjustments such as gamma increase or decrease, colorsaturation increase or decrease, brightness increase or decrease, orother alternation is desired, it will usually be best to begin suchalteration with an accurate linear light representation. For example, a1.1 gamma increase will produce a predictable presentation if the pixelvalues begin with a known transformation to exact linear light. Notethat linear light is an implicit gamma of 1.0 (i.e., this means that aunity exponent is the same as no “gamma” exponent).

Brightness of Mastering Screen

The current specification of white (maximum) brightness for a moviemastering screen is 14 fl, which is 40 cd/m² in metric. There iscurrently debate about the correct digital cinema white point. For filmprojection, the white point has been defined by SMPTE (Society of MotionPicture and Television Engineers) as being 5400 degrees Kelvin colortemperature with an asymmetric tolerance of +600 degrees and −400degrees.

For television mastering reference, the television screen white level isspecified to be 30 fl (85 cd/m²), although 22 fl (63 cd/m²) is thetypical brightness used. Television mastering is still done mostly usinga reference direct-view Cathode Ray Tube (CRT) display, usually about30″ (76 cm) in diagonal using a D65 correlated color temperature forwhite. Note that D65 is a daylight 6500 Kelvin equivalent, and has adefined spectrum. However, D65 is also defined as a “correlated colortemperature” in terms of CIE 1931 xy chromaticity. It is this CIE 1931xy chromaticity-dependent definition that is used to calibrate thereference white of digital television mastering displays.

It is common practice to adjust the red, green, and blue primary gainsto achieve the desired white, as specified using CIE 1931 xychromaticity. Thus, the “native white” of displays and projectors, withall gains at maximum, does not, in general, yield the intended whitebalance. Efficiency concerns for display and projectors attempt tooptimize the red, green, and blue primary adjustments required for thewhite balance such that they are as near as possible to their maximumvalues. In practice, one or two of the three primaries will usually bebetween 80% and 90% of whichever primary is set at maximum.

Digital cinema masters are usually ignored when re-mastering for digitalvideo, although there is no good reason that a proper transformationcould not be defined which would directly yield digital video. Themethods of the present invention yield such a proper transformation formost scenes.

One embodiment of the present invention uses (see below) a fourthprimary (if there are four or more) as a broad-spectrum white. This isoften beneficial to efficiency as well as being beneficial to coloraccuracy (as described below).

Aesthetic White Point

The aesthetic white point for a display or projection system can bedifferent from the calibration white point. For example, the SMPTE DC-28specification for white point for projector calibration for digitalcinema defines a greenish white point, defined in terms of CIE 1931 xychromaticity, to the green side of the daylight and black-body curves.However, the intent is that a mastering white near D60 be utilized.Further, there is discussion of allowing white points along the daylightcurve in the range of D55 through D60 up to D65, all at full 14 fl (40cd/m²) brightness.

If the maximum projector or display brightness criteria are ignored, itis clearly possible to have a very wide range of “aesthetic” whitepoints, independent of the calibration white points. Further, it wouldbe useful to use a broad spectrum white (see discussion below) toprovide the desired aesthetic white and gray, which will be lesssensitive to the choice of color matching function, and therefore lessdependent upon CIE 1931 xy chromaticity.

The human eye is highly adaptive to a wide range of white points.However, a “partial adaptation” mechanism of human vision will stillhave some approximate sense of the white point being used. This will betrue of the cinematographer, the colorist, and the director. Thus, oncean aesthetic white point is determined, it is useful to minimize driftin individual adaptation to the white point by providing a small regionof reference white. One technique useful in conjunction with the presentinvention is to provide a small white border around an image, in thedesired reference white, to help provide the white reference necessaryfor proper color adjustment. Note that “windows” borders on typicalpersonal computers provide this function (since window boundariestypically provide a small white border around an image), although theirwhite point is typically not adjustable. In addition to white, for darkscenes it will also be useful to utilize a darkened version of gray,having the same white color balance, to provide the neutral-gray colorsense for the reference viewers. In this way, both white reference forbright scenes, as well as gray reference (at various brightnesses) canbe present in the mastering room, preferably directly on the image, suchas a small border, to provide a white balance/gray balance neutralvisual reference. Thus, by maintaining this always-visible white or grayreference, a consistent relationship to the intended white balance/graybalance can be maintained.

Note that typical grays of 18% of peak white, and 10% of peak white, arecommonly used as a neutral (having the intended white point) colorreference.

It may also be useful to have various black-to-white gray intensityramps around an image, which provide a white, gray, and black referencesimultaneously. If such ramps have a defined gamma or a logarithmic rampfunction, they also help define the scene gamma, since the entireblack-to-white range is given across the ramp. If several such ramps areprovided on each edge (left, right, top, and bottom), it provides both awhite balance as well as a reference for white, gray, and black, whilemastering; an example of such an arrangement is shown in FIG. 6.

Note that when mastering in a darkened room, the absolute white sense isgradually lost via adaptation. For this reason, it is useful to have asmall presence of reference white, at the proper white point, somewhereon the display or projection screen at all times. Since such a patch ofwhite around the screen may interfere with the perception of the scene,it may also be useful to toggle the white patch (or gray orblack-gray-white ramp) on and off during mastering, to both provide areference as well as minimizing its interference with the aesthetics ofmastering scenes of still or moving images.

It is also possible to provide dark gray walls with proper color, and/ora white reference of the correct white point color, which are not partof the projection screen. However, the spectra are likely to bedifferent, and thus the choice of color matching functions would likelyalso be different. A wide angle set of color matching functions, such asCIE 1964 10-degree, or the 10-degree settings of CIE 170-1:2006 (seediscussion below) might be best, since off-screen white and grayreference colors are well outside the macular yellow pigment. Use ofon-screen white and gray reference provides the same white and grayreference spectra as is used within the image.

Regional Characterization

It can be useful to measure and spectrally characterize a screen atvarious locations for any or all combinations of primaries (although themaximum of each primary separately and together (white) will provide asubstantial portion of the necessary regional characterization). Forexample, most screens roll off in brightness near the edge; color andeven spectra may regionally alter for each primary and for white; andgamma may alter regionally (and/or linear and/or non-linear lightfunction of linear pixel values). The determined characterization maythen be conveyed to a presentation system to adjust display colors inaccordance with the present invention.

Some mastering projectors plus screens or displays attempt to correctfor regional variations. In such cases, the remaining lack of achievedcorrection may still be conveyed.

A measure of the regional black can also be beneficial, although thismay often best be done by inference (e.g., using 2% of white compared to1% of white in each region, as described above).

The measured information can be conveyed or provided as regionalweights, which are smoothly blended (e.g., using a 16 horizontally×8vertically grid of rectangles). However, if the spectra is regionallyaltered, then the spectra for each such region can be conveyed orprovided, and then blended via similar weights.

Regional Black

Most displays, and projectors with their associated screens, attempt tocalibrate for uniform black. However, this is generally difficult inpractice, since the ambient “black” of a display can vary regionallyover the display, both in terms of brightness and spectra. Accordingly,it is useful to convey or otherwise provide a measurement of theregionally-varying black. This can be conveyed spectrally, or via asimplification such as a regional amount of a common spectrum or via amore comprehensive regionally-varying spectral representation. Suchregionally-varying spectral representation can usually be conveyedadequately via spectrally-specified regions with smooth blending betweenthem. For example, a 16 horizontal by 8 vertical rectangularrepresentation of the black spectrum can be smoothly interpolated toyield the black spectral energy (corresponding to a zero pixel value) atevery point on the screen or display.

Directionality Characterization

In addition to regional characterization, with some types of displays(such as liquid crystal displays) and some times of screens (such ashigh-gain screens) there are also directional characteristics. Suchcharacteristics depend upon the angle of the screen to the viewer. Sincethe distance to the viewer affects the angle to various parts of thescreen, the absolute distance to the screen is also usually a factor indirectional characterization.

If there are multiple viewing locations, and the director sits in oneseat, the cinematographer in another, and the colorist in another, anydirectional affects will potentially alter the image as seen by eachperson. It may be useful, if there is much directional affect, toidentify the directional characteristics with respect to each key personwho is present during mastering any given scene.

Directionality with LCD displays is significant, affecting and alteringcolor, gamma (or pixel value function of linear light), and spectra, aswell as altering secondary affects (like room spill amount).Directionality with high gain or silver-3D-polarization-preservingscreens is also very significant. Thus, a measurement of the directionalcharacteristics, especially with respect to key viewers during masteringand/or any given presentation, is potentially valuable in properlyrendering accurate color.

Light Reflections on Projection Screens or Displays(Specular+Lambertian)

Light spilling from a room onto displays or projection screens is acombination of diffuse (lambertian) and reflective (specular) lightscatter back onto the viewer(s).

Reflections of light areas of the room are a factor (e.g., an exit signshowing up in the upper left of the screen as a faint green reflection)affecting what is seen on the screen. With high gain projection screens,or with the reflective cover glass (or plastic) of display screens, roomenvironment light can be directionally reflected, varying by region onthe screen.

On high gain screens and on reflective displays, colored specularreflections of objects and lights in the room also form a factor. Forexample, a high gain silvered polarization-preserving screen (typicallyused for 3-D projection) will reflect a brightly-colored shirt or blouseat the reflection location (which is usually the equal-angle reflection,as in minors, but dispersed somewhat by the screen). Use of low gain(between 0.8 and 1.3) mastering screens and matte unreflective displaysfor mastering can help minimize such specular reflections.

Lambertian (smooth) light spill often occurs in both projection anddisplay environments. One common mastering theatre has bright orangeseats, for example, which spill orange light back onto the screen(further, the bottom of the screen receives more orange spill than thetop). Someone wearing a brightly colored shirt or blouse (such as yellowor green) sitting in front of the projection screen or the display isalso likely to spill any color light, including white, from the screenback onto the screen in their shirt's color.

Note that there may be changes over time, such as someone turning on asmall desk lamp while mastering, even though the calibration was madewith the desk lamp off.

Optimum viewing occurs when the environment is dark neutral gray orblack, although most viewing environments are gray or beige in order toprovide pleasant surroundings. Thus, room spill is a substantial issue.Note that room light spill back onto the screen is partially a functionof who and what is in the room, and is also partly a function of whatcolors are on the screen, what their spectra is, and how bright theyare. Such issues are also a function of where on the screen the light isbeing emitted, and where in the room, the bright colors (like the blouseor shirt) are located.

Note that such issues will also be a factor in home viewing, officeviewing, and theater viewing.

Note that mastering and presentation display or projector+screenlambertian and specular light spill and reflections are generallyignored in current practice, but it is better to characterize them, andpossibly attempt to remove them, if feasible.

Accordingly, it is best to control such issues by either includingmeasurements of such changes as information about the mastering room, orby not allowing variations in the room lighting environment duringmastering compared to when calibration of the room was measured. Forexample, wearing dark clothing while mastering is desirable.

Flare Correction

Most displays and projectors spill light from each pixel onto otherregions of the screen, with most light being spilled into nearby pixels.The ratio of black to white is often specified in terms of “sequential”contrast, where an entire screen of black is followed by an entirescreen of white, or alternatively by “simultaneous” contrast, wherewhite and black checkerboard squares appear on the screen together.Typical sequential contrast is anywhere from 200:1 up to 2500:1, whereassimultaneous contrast is typically around 100:1. Thus, light spill frombright image areas onto adjacent dark image areas is significant.

The light spill concept is sometimes called “flare”, and algorithms for“flare correction” attempt to correct for this localized regional lightspill.

Lambertian diffuse light spill into a white room will illuminate theentire screen, whereas localized light spill due to glass reflectionswithin a display screen will only spill onto nearby regions. Thus, anycorrection for flare must take into account the proportions of bothlocalized (regional) and wide area light spill.

Mastering and presentation display or projector+screen flare isgenerally ignored in current practice, but it is better to characterizeit, and possibly attempt to remove it, if feasible.

Note that flare correction can affect the pixels in the master ifapplied to the mastered pixels. Alternatively, a characterization ofmastering display or projector flare can be sent with the image, suchthat the pixels are not “pre-processed” with flare correction. If themastering display or projector has relatively low flare, characterizingthe flare, but not correcting for it, is likely to be most useful. Uponpresentation, if a given display's or projection screen's flare differssubstantially from the flare present in the mastering room (such asbeing much greater), then it will often be desirable to attempt toreduce or remove the flare during such presentation using flarecorrection based upon the conveyed characterization of mastering roomflare and measurement of the presentation room flare.

Screen-to-Room-to-Elsewhere-on-the-Screen

Light displayed on one portion of a screen affects other portions of thescreen, For example, a bright white patch on a screen's upper rightilluminates the middle with some amount and spectra, as well as thelower left, and upper left, and lower right, etc. This effect isgenerally ignored in current practice, but should not be. This is one ofthe components of flare, as well as being a component of lambertian(non-reflective) light spill. Accordingly, conveying a measurement ofthis effect for later use to correct a displayed image is useful.

Inter-Pixel Bright-Image Darkening Influence

Historically, with CRT's, often there was poor high voltage power supplyregulation when large areas of high brightness were displayed, such thata particular pixel value would generate less light if large regions ofthe screen were bright. This affect is also active with some plasmadisplays, which use dedicated circuits to reduce overall brightness whenlarge amounts of the screen are bright, in order to better preserve thelife of the display unit.

This affect is generally ignored in current practice, and may usefullybe ignored with some display technologies, such as DLP projectors andLCD direct-view displays, which do not exhibit this effect. However,with displays or projectors which have this issue, small patches ofcolor, white, or gray on a dark surround can be used for makingmeasurements of this effect. The affect of large areas of highbrightness can then be modeled independently, and would not otherwiseinterfere with necessary measurements of spectral energy from variouspixel values (inside small patches).

Ambient Surround During Mastering

In addition to light spill back onto a screen, the eye is influenced bythe colors within the surrounding room. There will be influence from thecolor of walls, furniture, ceiling, etc., around the screen. The colorsof walls behind or surrounding a screen may not result in light on thescreen, but will provide light into the eye. While motion pictureprojection for mastering and exhibition intends use of a dark surround,television systems intend the use of a gray background during mastering(although background brightness and color varies widely in the home andoffice viewing environment). For example, there are television masteringspecifications for image surround brightness and color which specify D65correlated color temperature (i.e., using CIE 1931 chromaticity for D65)background at 10% of peak white as the standard for video mastering at30 fl (although 22 fl is typical in practice).

If there is such a known ambient surround during mastering, its colorcan usefully be measured and conveyed along with other masteringspecifications. The color and brightness are most useful when conveyedas spectral energy, but can also be usefully conveyed using luminanceand chromaticity. Given that ambient surround will image well outsidethe macular yellow pigment, the use of CIE 1964 10-degree color matchingfunctions, or the CIE170-1:2006 cone fundamentals set at 10-degrees, aremore appropriate than CIE 1931, which is based upon 2-degree color patchmatching. Providing the spectral energy for the surround provides themost useful information. Given that the surround is likely to not beuniform, a measurement of regions (such as 16 regions surrounding thescreen) will be more useful than an average surround result, althoughthere are as yet no models for regional variations in surround color andbrightness. A general model of a relatively constant surround does existwithin some Color Appearance Models, but it is most often modeled usingCIE 1931 luminance and chromaticity, or equivalently using CIE 1931 XYZtristimulus values. As such color appearance models improve, it islikely that they can begin to take advantage of spectral energy andpossibly regional variation information for the visible surround.

Concept of Mastering Beyond Available Gamut and/or Dynamic Range

It may be the intent for a master to create more saturated colors thanare available on the mastering display or projection screen. It mayfurther be the intent for the master to create brighter whites andbrighter colors than are available on the mastering display orprojection screen. Such intent can be conveyed with the master, andapplied appropriately when the presentation projector or displayprovides for an extended color and brightness range.

Levels of Characterization

Levels of characterization apply not only to the mastering room, but toany presentation environment having a display or projector.

As described above, at the most basic level of characterization, justRGB values as a function of linear light and their spectra can bedetermined and conveyed.

A useful next level of characterization can add the spectrum energy ofblack.

A useful next level is to add regional functions of RGB values.

All of the characterizations and/or corrections of the transformationfrom pixel values to emitted light, such as flare correction orcharacterization, can be usefully added in logical steps, based upon thedescriptions above.

Characterization Versus Correction

For some of the above issues, such as flare correction, such correctionwould either be applied or not applied during presentation on any givendisplay or projector+screen, based upon the flare properties of thatdisplay or projector+screen. Thus, for presentation, the use or absenceof any given correction is all that utilizes the characterization ofthat device. The characterization for the presentation display orprojector+screen serves no other purpose than as a means for correction.

However, for mastering, a characterization of the flare, which impliesthe means of flare correction, can be considered independently ofwhether flare correction is actually applied to pixel values or not. Theuseful cases are: (1) a characterization of the flare without applyingany flare correction; this provides information about the inherent flareseen on the reference display or projection screen when the image wasmastered; (2) a characterization and a correction for the flare, whereinthe correction is not applied to the pixel values of the master, but isapplied when the pixel values are sent to the mastering screen); and (3)a characterization and a correction for the flare, wherein the pixelvalues of the master have had the flare correction applied.

In the third case, it would be potentially useful to be able to invertthe flare correction, in order to be able to recover the uncorrectedpixels, as with the uncorrected form of the pixels used in the masterfor the second case. However, although some of the processes describedabove have practical inversions, a complex process such as flarecorrection may not be easily invertible.

Thus it is potentially useful to convey which of these cases ofcorrection and/or characterization are applied with respect to eachissue described above. Further, if a correction is present in the pixeldata, it is potentially useful to convey the inverse process to yielduncorrected pixels, if such an inverse process is feasible to describeand implement. It may also sometimes be best to send both corrected anduncorrected pixels, which is the only way to convey both cases when theinversion process is impractical.

Generic Characterization

It is also likely to be useful for manufacturers of reference displaysand projectors (or neutral third parties, such as standards groups andindustry associations) to provide a full characterization for a genericmodel. A generic room must also be assumed. In the case of projectors, ageneric screen must also be assumed. Alternatively, additional genericrooms (e.g., changes in ambient lighting) and projection screens (e.g.,changes in screen gain) can be modeled. This data can be made availableto anyone using that model of display or projector, in order to providea more comprehensive characterization when only a minimalcharacterization is available for a specific model in a specific room,or in order to provide a characterization where a specific model is notcharacterized or where a specific room or screen is not characterized.

Certain common models of monitor and projector are known to be mostpopular for use in mastering. Certain facilities are also known to usecertain models, and such information can be useful in conjunction with ageneric characterization.

The use of generic characterization(s) according to model of display orprojector can also be extended to end-user presentations, in addition toprofessional mastering environments. In end-user presentation, themanufacturer, or a neutral third party, can afford to perform areasonably extensive characterization, or set of characterizations, evenif the end unit price for the display or projector is low, such as whenthere are large volumes of units sold.

For historical television masters, certain models and brands of CathodeRay Tube (CRT) were most common for PAL, SECAM, and NTSC color masteringduring various decades. Certain mastering facilities were also known tohave used certain models and brands during certain periods. The spectraof these models and brands can therefore be utilized, although allowanceshould be made for some variation in calibration. Thus, such genericcalibrations may be useful, but should not be completely relied upon inall cases.

Simple Versus Cross-Color Characterization

The ideal characterization would have independent primaries (usuallyred, green, and blue, but optionally including additional primaries asdescribed herein) wherein each primary would have a known mapping (suchas a pure gamma or video gamma) to the linear amount of light emitted ineach primary's corresponding spectrum. This has been true at photopiclight levels with high-end digital projectors, but is unlikely to betrue near black with any projector or display (due to the addition ofthe ambient room spectrum).

Many displays and projectors, and the graphics cards used to send pixelsto such displays and projectors, perform processing having intentionalcross terms and matrix processing. Sometimes the matrix multiply processis performed correctly on linear light pixel values, and sometimes thematrix multiply process is performed incorrectly using non-linear (e.g.,gamma-adjusted) pixel values. Other processing, such as chromaresolution reduction horizontally and/or vertically (e.g., 4:2:2 or4:2:0 UV reduction in YUV representations) may be applied, and it iscommon practice to apply the resolution reduction filters and matrixmultiply process to non-linear pixel values. This process cannot beinverted, due to the U and V resolution reduction.

Further, matrix processing, even if performed correctly on linear pixelvalues, will usually cause an amount of a given primary (for example,red, from red, green, and blue), to weight some amount of one or more ofthe other primaries (for example, green and blue as a function of red).If such processes are defined, they can sometimes be properly inverted.If such processes are not defined, then a full cross-colorcharacterization is needed (involving a 3-D cross-color lookup tablewith 3-D interpolation in the case of three primaries). Such acharacterization will yield the spectrum emitted by each combination ofred, green, and blue pixel values (and optionally more than threeprimaries). Such a cross-color characterization has long been used bythe present inventor (beginning in the 1970's) to characterize cameranegative film, reversal film, and print film for motion pictures andstill images with red, green, and blue inputs and outputs.

In the present invention, however, each combination of high-order bitsof all primaries is preferably used to look up an emission spectrum(instead of just red, green, and blue outputs). The interpolation of thelow order bits will be applied to interpolate the spectrum. For example,if the spectrum is defined at a precision of 5 nm, 4 nm, 2 nm, or 1 nm,there will be 80, 100, 200, or 400 values, respectively, to interpolate,spanning the range of 380 nm to 780 nm (sometimes stopping at a reducedrange of 390 nm to 730 nm). Each wavelength can be interpolatedindependently. Thus, the historical (since the 1970's) “cross-color” and“3-dimensional lookup and interpolation” concepts are seen as 3-primaryto 3-primary processes (or N-primary to N-primary, if they were to havebeen extended to more than three primaries). However, the presentinvention requires that the high order bits of 3-primaries (or Nprimaries if more than three) be mapped to spectra, and that the spectrabe 3-dimensionally (or N dimensionally) interpolated. Again, this can bedone independently for each wavelength. A 3-dimensional cross-colorlookup table and corresponding interpolation can be seen as a3-wavelength degenerate case (albeit with wide-bands at thosewavelengths) of this concept (remember that the visible spectrum istypically defined for every 1 nm, 2 nm, 4 nm, or 5 nm between 380 nm and780 nm). Thus, in this example, an N-primary mapping to spectralemission energy will result in 80, 100, 200, or 400 values for 5 nm, 4nm, 2 nm, or 1 nm, respectively.

Such a cross-color spectral characterization provides the necessarycharacterization for those displays and projectors+screens which cannototherwise be cleanly defined in terms of independent spectral channels.Even for cleanly defined displays and projectors, such acharacterization may be appropriate at low light levels in order toproperly define the spectrum and amount of light emitted for dark pixelvalues. Using such a characterization, the spectrum of the display orprojector+screen can be recreated. This characterization process can beapplied to both mastering and presentation.

The spectral lookup table is usually implemented using an enumeration ofall values of some number of high order bits (e.g., the high three,four, or five bits). If there are three primaries, then the table ofspectra is a 3-D table. If there are “N” primaries, then the table ofspectra will be “N” dimensional. The low order bits are usuallyinterpolated using piece-wise linear interpolation, although smoothercurves (such as a spline fit) may be needed if there are insufficienthigh order bits to provide for piecewise linearity (making sure thatnegative energy values are clipped to zero energy). An example might bethree color primaries of red, green, and blue, plus broad-spectrumwhite, thus being four “primaries”. If four high order bits areenumerated for all sixteen possible values, then the total size of thelookup table is four bits times four primaries=16 bits=65536 (i.e., 64k) table entries, each having between 80 and 400 entries (for 5 nm to 1nm spectral precision, respectively).

It is best to obtain a display or projector setting which minimizes, orpractically eliminates (except for darkest black) the cross-primaryspectral energy. With a defined mapping of pixel values to linear lightfor each primary, the spectrum of emitted light can be simply obtainedby using the pixel's primary values to linearly weight the spectrum ofeach primary, and then summing the resulting weighted primary spectra.This greatly simplifies the determination of the emitted spectrum as afunction of primary color pixel values.

If cross-primary terms cannot be disabled, then the next best situationwould be to have a known inversion, which can be implemented by astraightforward set of mathematical steps. In the absence of a practicaland defined inversion, however, a cross-primary characterization willusually be required.

A hybrid is also possible, where a cross-primary table and interpolationis used for the lowest portion of brightness, such as below 3% of peakdisplay or projector+screen white. For example, using red, green, blueand broad spectrum white, with three high bits of zero for all fourprimaries, but all other bits one (for all four primaries), might yield3% of peak display or projector+screen white, then the next three bits(below the high three bits of zero) can be enumerated in each of theprimaries with 12-bits (being 3 bits times 4 primaries) and interpolatedto yield the dark spectra (from 3% of peak white down to maximum black)as a function of red, green, blue and broad spectrum white pixel values(wherein each has high order 3 bits of zero).

Automating the Characterization of Displays and Projectors+Screens

As technology advances and costs of complex electro-optics drop, it willbecome feasible to place inexpensive spectral radiometers in cameras,projectors and/or displays. Even simple red, green, blue cameras, whichare becoming very inexpensive, provide substantial color and dynamicrange information (using their red, green, and blue sensing spectra).Such cameras, and eventually even spectral radiometers, can be used notonly for imaging, but for calibrating cameras, projector+screens, anddisplays.

It is feasible to automate the process of reference projector orreference monitor characterization. For example, the automated processcould be run every morning for a particular reference setup.

It is also feasible to automate the process for characterizingpresentation monitors or projector+screens. Some calibration can beaccomplished on the fly while the display or projector+screen operates,by examining the spectra being presented for various pixel values whichare occurring.

It is also practical to automate the examination and correction for theambient surround during viewing, and prior to viewing.

While running automated characterization processes, after operating acalibration camera on a variety of useful moving images for a fewminutes, a completely characterization of the behavior of the display,monitor, or projector+screen can be obtained in accordance with thedescription herein, which can then be used to build an accurate colormodel for the display, monitor, or projector+screen.

This methodology can be applied for reference monitoring and/orprojection, as well as for presentation viewing, display, and/orprojection.

It is further possible to apply electronic computation and sensors toevery display and projector+screen to improve color fidelity.Measurements of the display for a variety of useful pixel colors invarious regions of the display, as well as measurements of the surround,can be made. More than three color channels can also become commonplacefor many types of display and projection technologies as costs decline,such as by adding a clear segment to a red, green, blue color wheel(which is already done with computer business presentation projectors).Greater dynamic range and higher bit depths per pixel are also likely togradually become economical over time. Higher resolution can become moreavailable and more affordable. All this will lead to the need for themethodologies and systems of the present invention in order to build ascientific and engineering basis for fully using these new capabilities,and for providing improved accuracy and precision compared to today.

Optimizing Orthogonality

Orthogonality is defined as being a property of mathematical functionsor matrices wherein most or all cross-terms are zero. The search forindependent primaries, which can each weight independent spectra, is atype of orthogonality. Even if a device exhibits cross-terms betweenprimaries, and can only be characterized by a full high-order lookup andinterpolation, it is possible to process the data to determine ifportions of the ranges of values of primaries are orthogonal (and thushave no cross-terms within those ranges). If all of the cross-termswithin such sub-range can be eliminated, the elimination process isequivalent to computing a transformation to a diagonal matrix,(typically implemented using eigenvalues and eigenvectors) using matrixor functional algebra, wherein the terms of the diagonal matrix thenbecome the weights of each primary. For the matrix approach, theanalytical process for determining the diagonal matrix, if it exists, isa similar type of computational process to matrix inversion. There arenumerous mathematical toolkits which have been developed for determiningif all or part of a diagonal matrix exists. The primaries after such atransformation are then orthogonal, and each primary then becomes amember of a mathematical “basis set”. (Note however, that the terms“spectrum” and “spectral radius” as used with eigensystems are notrelated in their meaning to light spectrum as used herein.)

Even partial orthogonality is beneficial. For example, a transformed redprimary might be isolated such that it weights an unchanging spectrum,even though green and blue interact with each other's spectra viacross-terms. For another example, the red primary might be sufficientlyindependent above a dark value (for example, 5% of peak white) such thatit can be used to weight an unchanging spectrum for values between thatvalue (e.g., 5%) and 100% of peak white, yet the red must interacthaving cross terms with green and blue below that value (e.g., 5%).

Matrix processing wherein any regions of the matrix have zero valuesafter transformation can help simplify subsequent processing, as well ashelp in characterization and understanding of the relationship betweenthe primary values and between the primary values and the resultingemitted spectra. It may also be useful to have a small threshold, belowwhich any given cross term is set to zero (and thus ignored).

Thus, if any significant portion of the range of values of any of one ormore of the primaries can be isolated (possibly via transformation) toweight an unchanging (or minimally changing) spectrum, the rangerequiring 3-dimensional or N-dimensional spectral lookup andinterpolation can be reduced (often very substantially reduced). Also,it is common for the spectral energy of color primaries to be limited toonly a portion of the 380 nm to 780 nm range, with zero energy at otherportions. For example, a blue primary may have positive energy between380 nm and 570 nm, but zero energy between 570 nm and 780 nm. In suchcases, spectral regions with no energy for one or more primaries can beutilized to simplify processing for those regions with respect to otherprimaries. Cross-color terms may therefore be zero for certain spectralranges, and nonzero for others. Thus, if the spectrum is divided intoranges, each range may have more or less complexity when going fromamounts of primaries to the resulting emission spectrum. For example,the range of spectrum between 380 nm and 570 nm may require3-dimensional or N-dimensional cross-color lookup and interpolation,whereas the range between 570 nm and 780 nm may be accurately andprecisely determined with a simple linear matrix or even via simplelinearized primary weights of independent primary spectra. Thus, whileconceptually the characterization may differ for every 1 nm of spectrum,in practice ranges of wavelengths may be significantly simplified by thepossibility of complete or partial matrix diagonalization for suchranges.

Given that the result of a combination of amounts of primaries is aspectrum, the process of diagonalization must be applied to amounts ofwavelengths of spectra. The spectra may not lend themselves to suchdiagonalization. In the limit, the diagonalization processing (ifpossible) can be applied independently to each wavelength (e.g., each 1nm). If such diagonalization yields a common result for all wavelengths,then full simplification into independent primary amounts of spectra hasbeen achieved. Similarly, regions of wavelengths can be simplified viafull or partial diagonalization by having common diagonalization resultsover a range of wavelengths (e.g., 570 nm to 780 nm). The general case,however, may find slightly different diagonalization processes at everywavelength. In such a case, the complexity of the diagonalizationprocesses should be weighed against the use of a 3-dimensional orN-dimensional high-order-bits lookup and interpolation.

The goal of all such processing is to find the simplest practical methodby which a given combination of values of primaries can yield thecorresponding spectral emission. Within the constraints of interpolationaccuracy, the 3-dimensional or N-dimensional high-order-bits lookup andinterpolation provides the resulting spectrum without requiring furthersimplification. Many common cases may require 3-dimensional orN-dimensional high-order-bits lookup and interpolation, unless both themastering and presentation display or projector+screen can be greatlysimplified, such as by being the linear sum of linear amounts of spectraof three or more primaries. However, there also commonly exist suchsimplified scenarios above black (e.g., between 2% of peak white andpeak white). Which implementation methods to use must be chosen withknowledge of primary independence versus cross-primary-color spectralinfluences. Any implementation that can be practically implemented, withdefined accuracy, can yield the resulting spectrum from a givencombination of red, green, blue, and perhaps additional primaries.

Note that for digital displays and projectors, the search fororthogonality and diagonal matrix (or even the search fortransformations yielding some zero matrix terms) is equivalent to thesearch for the inversion of any cross-term processing which may beapplied (but may also be hidden) within a graphics card's and/or adisplay's or projector's internal processors. If any orthogonalprimaries exist anywhere within the display or projector (even ifhidden, undocumented and inaccessible), for all or part of their range,the diagonal matrix search will find them. The resulting transformationmatrix (or function), if it exists for all or part of the range ofprimary values, will then accurately and precisely describe the forwardand inverse cross-term processing (for that range of primary values).

Note that matrix diagonalization (full or partial) is a linear process,as are all matrix processes, and that any non-linearities within theprocesses inside the graphics card, display or projector will inhibitdiagonalization. Thus, it is valuable to gain an understanding of how tocorrectly linearize any pixel values and their resulting relationship tolinear light, whenever possible. Non-linear relationships will inhibitthe degree to which partial or full diagonalization can be achieved(thus requiring the 3-dimensional or N-dimensional lookup andinterpolation of spectra, which can be applied directly to lookup andinterpolate using non-linear pixel values).

Interpolating Variations in Spectral Precision

If one spectral measurement has one precision, for example 2 nm, and acolor matching function has a different precision, for example 5 nm,then they will need to be interpolated in order to be integratedtogether. The simplest approach is piecewise linearly interpolation. Forexample, the intervening four values to linearly interpolate 5 nm datato 1 nm data are obtained using weights of 0.2, 0.4, 0.6, and 0.8 and0.8, 0.6, 0.4 and 0.2 applied to adjacent wavelength energies. For 2 nminterpolation to 1 nm, linear interpolation only requires that an equalaverage weight of 0.5 be used for each wavelength neighbor at 2 nmintervals. Once both spectral data are interpolated to 1 nm, they can beintegrated together by multiplying them and then summing over allwavelengths.

A smoother interpolation can also be used. Various spline fits arecommonly used to smoothly interpolate spectral data, including“official” CIE recommended practices; other published recommendationsexist. Care should be taken to clip negative values (if any) to zerowhen using higher-order (beyond linear) interpolation methods.

It is simplest if all spectral emission data and all color matchingfunctions are specified at a common wavelength precision. However, ifvariations in precision are anticipated, then the precision must besignaled or implicitly known by context.

Augmenting Uniformly-Sampled Wavelength Energy with Spectral SpikeEnergy

A typical spectral energy sampling interpolation filter is usuallyapplied optically on a wavelength-selecting grating within many spectralradiometers. Such a filter is usually “triangular” in its intended shape(as a function of wavelength), but is often a shape mixture oftriangular and Gaussian in practice. Such a filter will spread some theenergy of a spectral spike into several nearby wavelengths, beyond theimmediate neighbors. An improvement of the present invention is toidentify spectral energy spikes within the spectrum, and then to removesome or all of their energy from the uniformly-sampled wavelengthspectrum. A list of the remaining spikes, and their corresponding energy(at least the portion which was not removed from the uniformly-sampledwavelength spectrum) can augment the uniform-wavelength spectrum. Forexample, if there are wavelength spikes corresponding to mercury (in amercury vapor lamp) at 405 nm, 436 nm, 546 nm, and 578 nm, these can begiven in a list with the corresponding energy of each, in addition tothe uniform-wavelength spectral energy. Note that the wavelengths ofspectral emission spikes are often known with very high accuracy (to atiny fraction of a nanometer).

With display and projector+screen emission spectra, large spectralenergy spikes should be avoided. However, if spectral spikes arepresent, they can be conveyed more accurately and precisely by partiallyor completely separating their energy from the broad uniformly-sampledspectral energy.

The color matching function spectral weightings are usually smooth. Oncethese color matching functions are smoothly interpolated to anyarbitrary fine wavelength precision, and/or to specific precise andaccurate wavelengths, specific spectral spikes can then be integratedprecisely and accurately. The resulting values can be summed into theintegral obtained from the uniform-wavelength spectral energy (havingsome or all of the spike energy removed) to obtain increased accuracywhen applying any color matching function (via integration) to thisspike-augmented spectral energy.

Note that spectral radiometers typically have limited accuracy in theirwavelength calibration, usually at about ±1 nm (although this accuracyvaries per instrument type and model). Further, this calibration mayvary over time and normal jostling of the spectral radiometer. Such avariation is most problematic relative to spectral spikes. Further, thespecific interpolation function may not yield completely uniform energyin spectral radiometers as the wavelength calibration varies slightly,due to imperfections in the optical (and possibly also digital) spectralsmoothing filter(s). Thus, the segregation of spectral spike energy, asdescribed here, can significantly reduce errors in spectral energyreadings from spectral radiometers.

Augmenting Mastered Spectra with True Scene Spectra

Some regions of some original scenes may have an approximately knownspectrum, or a spectrum which can be defined by a few simple parameters.For example, if a region of an image is know to be portraying apine-wood fire in the dark, the spectral emission energy of a pine-woodfire is approximately known, and can be identified for the region of theflames. A warm-fluorescent overhead light may have a defined spectrum,including mercury green spikes and other characteristic spectral spikes.A daylight sky may have a defined color temperature, e.g., D50, whichcan define the spectrum of a normal sky at that color temperature. Thus,it is possible to augment any region of any image with spectralidentifications which can be used to create natural actual spectra in aprecise way, thus avoiding any dependence upon the spectra of theprimaries of the mastering display or projector+screen. Note that somesuch spectra, such as flames and fluorescent lights, will naturally havespectral spikes, which can be defined separately as described above.

In addition, the color matching functions which are used by keypersonnel to view the mastering display or projector+screen can definethe brightness level and chroma and hue of any color, including anatural color. Using these color matching functions, parametric naturalcolors, such as daylight as a function of color temperature, can beapplied at the proper parameter value to yield the proper colorappearance on the mastering display or projector+screen as viewed by thekey person(s). Thus, a region (e.g., sky or flames) would appear correcton the mastering display when integrated with the color matchingfunctions for one or more key people, but the definition couldalternatively reference actual spectra (such as sky at a particularcolor temperature, or flames in a given circumstance, etc.). In thisway, one or more spectral alternatives can be provided in addition tothe colors described using the spectrum being presented on the masteringdisplay or projector+screen.

Additionally, or alternatively, regions of an image can be initializedto defined spectra (e.g., daylight or flames) and then color balancedusing a simulated color adjustment filter to alter the balance ofspectral energy at various wavelengths. If colors are adjusted in thismanner for defined-spectra objects (reflective and/or glowing) in thescene, then the altered spectrum can then be conveyed for those regions.This altered spectrum (like a redder fire or a bluer sky) can beconveyed directly, or via the natural spectrum plus the simulated coloradjustment filter transmission as a function of wavelength. Thisoriginal-object-based spectrum (possibly adjusted via a simulated colorfilter) for one or more regions in the image optionally can be conveyedin addition to the spectrum energy emitted by the sum of primaries fromthe mastering display or projector+screen. The amounts of the emissionprimaries from the mastering display or projector+screen are required todefine the shading and color variations of such regions which havedefined spectral information as augmentation. In the absence of thenormal pixels from the primaries, the entire spectrum for such augmentedregion would need to potentially be varied per pixel, which is likely tobe impractical (although possible under some useful circumstances). Amore practical approach which can often be used is a smooth variation ofspectra via regional weighted variation. While this may not work fordetailed rapidly moving objects such as flames, it can work well forsmooth regions of an image such as a sky. The object or light spectralinformation from regions would typically represent augmentationinformation, augmenting pixels defined in the mastering primaries, foroptional use during presentation. However, regional spectral definitioncan be useful as a sole definition of the color of a region in someuseful cases.

If spectral radiometric measurements are available for objects, objectsurfaces, radiant objects (like fire), or light sources within theoriginal scene, any such measured objects or lights can be processed inthis way. Thus, the known original spectra can be adjusted as if theentire spectrum is filtered and modulated by appropriate amounts ofmastered pixels (in the mastering primaries), and then the resultresampled using appropriate color matching functions and the spectralemission of the primaries of the presentation display orprojector+screen.

This technique can be used for improving the accuracy and precision ofcolor (both surfaces and lights), in addition to, or alternatively to,providing means to portray scene colors in creative ways. Displays orprojectors+screens with extended color range or brightness range orextra color primaries can also benefit from such extended definitions ofcolor spectra.

Augmented Spectral Effects and Other Effects

Additional spectral effects can be handled in a similar manner bydefining regions having such effects. For example, if a surface in theimage has fluorescent properties, then the region of that surface can bedefined in terms of the spectral energy of visible light that is emittedgiven an amount of incident ultra-violet light. In this way, day-glowcolors, optically-brightened paper and cloth, and other visual affectscan be recreated during presentation with a specified ultra-violet lightamount and corresponding defined spectral fluorescence.

In three-dimensional stereoscopic image presentation, iridescence can bedefined for a region (like butterfly wings), such that the colors andillumination for each eye differ and sparkle and fluoresce as they do inreality.

Other spectral interactions of surfaces and lights can be reproducedduring presentation based upon defining a region, and specifying therules for the emission spectrum from the region. The color matchingfunctions can then be applied to the resulting spectrum and the emissionspectrum from the display or projector+screen (using matrix inversion)to create the intended appearance and the intended visual color andbrightness effects which cannot otherwise be created solely bytransforming color primary spectra.

This technique can also be used for accuracy and precision of color forthese special situations (both surfaces and lights), in addition to, oralternatively to, providing means to portray scene colors in creativeways. Displays or projectors+screens with extended color range orbrightness range or extra color primaries can also benefit from suchextended definitions of color spectral effects.

Some such effects need not alter spectra but may define extensions todynamic range or other useful alterations. For example, a sparklingdress will be limited in the maximum brightness of the sparkles to themaximum brightness on the mastering display or projector+screen.However, it the region of the dress is defined as having sparkle flashesto a much higher brightness than is available on the mastering displayor projector+screen, any presentation having a higher brightnesscapability can render the sparkles with more accurate bright flashes,even if the other scene brightnesses and colors are directly related tothe pixel brightnesses used during mastering. In addition, some forms ofsparkling material may be spectrally colorful, and may thus generate notonly very bright flashes, but also very colorful flashes, which mayextend beyond the maximum saturated color range of the mastering displayor projector+screen. Such extended color brightness for pure saturatedcolors during the sparkle flashes can be extended on presentationdisplays or projectors+screens having such an extended color range forhigh brightness saturated colors. This extended presentation range isenabled by defining the region as having such extended color range, evenif such bright pure saturated colors are not defined in the masteredpixels (due to limitations of bright pure saturated color range on themastering display or projector+screen). The extended range of colors andbrightnesses for a sparkling dress can be defined using additionalinformation to guide the presentation, in addition to an outlinedefining the region. If the actual range of brightness information isknown, it can be added, although it may also sometimes be useful toindicate that the sparkles should use the full range of the presentationdisplay.

Note that displayed sparkles may have an expanded size (e.g., due tolens flare and halation) to cover a larger area than the size of theoriginal sparkles. Alternatively, it may be desirable in somecircumstances to intentionally expand the size of sparkles in order toemit more light (due to the larger area of white) for each sparkle.However, another option is to intentionally reduce the size of sparkles,so that they have a fully bright extended dynamic range color andbrightness, but only in a single pixel, or other useful smaller (orlarger) size than the original. This concept can be further extendedwhen a mastered image is upsized and presented on a display havinghigher resolution than the master, such that the final displayresolution is used for the single-pixel (or other reduced size) brightsparkles. The size of sparkles significantly affects their textureappearance, in addition to their brightness and color.

There are many such useful resolution-altering effects, color-alteringeffects, brightness-altering effects, contrast-altering effects, andother defined effects (like fluorescence) which can be applied toregions using extensions beyond the limits of pixel values within themastering spectral range and beyond the mastering dynamic range.

Many useful types of such regionally-defined extensions can be added tothe pixel values (which define amounts of the mastering spectralemission energy). The two primary uses of such augmentation are to allowincreased precision and accuracy as well as to extend the scenepresentation information beyond the range of the mastering display orprojector+screen.

Extended Range Mastering with Deferred Print Emulation

Some cameras, such as the Thomson Viper Filmstream camera, providewide-dynamic range information during original photography. While suchinformation is likely to be interpreted (via color adjustment) and thedynamic range is likely to be collapsed (via film print emulation), theoriginal extended dynamic range is available in the original wide-rangepixel values. Such wide-range original values may be conveyed orprovided for use. One way of conveying them is to specify coloradjustments and print simulation during mastering for use inpresentation, but to convey the actual original pixels. Another methoduseful in particular with the present invention is to use atransformation to linear light in a floating point numericrepresentation (retaining the wide dynamic range), using OpenExr 16-bithalf-float values, with scale factors for color and exposure balance,and a definition of the non-linear print simulation to use for eachpresentation environment (there may be several, and/or adjustmentguidance may be provided).

If such wide dynamic range master pixels are retained, together withtheir defined print simulation and color adjustments used with themastering display or projector+screen, the mastered image can berecreated. Further, in alternative presentation environments, definedalterations to the print emulation can be utilized, as appropriate. Forexample, a wide dynamic range display capable of higher brightness forwhite and darker blacks compared to the mastering display orprojector+screen can use more of the original dynamic range forpresentation than was available during mastering. The presentation on awide dynamic range display or projector+screen may also utilize a highereffective gamma, deeper blacks, and allow peak white highlights to gomuch higher above normal scene white.

The print emulator can also be sensitive to the presentation display'sor projector+screen's color range (e.g., color gamut and maximumsaturated color range), and the ambient surround, and other factorswhich affect appearance to optimize presentation. The bright saturatedcolors may be extended for such presentation without adding anysaturation to normal colors (such as face colors, wood textures, wallpaint, and other typically low saturation colors which usually requireaccuracy and precision). However, saturated colors on flowers in thesame scene may be set to utilize the full color saturation range of thepresentation display or projector+screen.

Such optimization can aim for accuracy and precision, or it may aim formaximum aesthetic affects, or a combination. Further, personalvariations and adjustments may be allowed, possibly within definedranges (or possibly disallowed). For example, some people like scenespresented in warmer (yellower) color temperatures (such as is seen undertungsten lighting), whereas other people like bluer scenes and bluerwhites (cooler color temperatures). Some such preferences could beallowed, while others may be inhibited by the creators of the image orshow since such alterations may interfere with the intended aesthetics(which may be intentionally discomforting to convey important moodthemes, or which may be intentionally comforting, and should nottherefore be subjected to alterations due to personal viewingpreferences).

Some types of shows are likely to best use accurate and precise scenecolor (like sports) versus other types of shows which may emphasizestage sets and face makeup (usually for aesthetically pleasingappearance) and yet other modified (unnatural) appearance forstorytelling and mood (sometimes aesthetically pleasing, and sometimesintentionally disturbing).

In addition to, or as an alternative to providing the wide range pixelsand one or more static or parametric transformations (typicallyincluding print emulation), it is also possible to send the wide rangepixels in addition to the pixels that have been mastered (usually with aprint emulation) on the mastering display or projector+screen. The imagecreated on the mastering display or projector+screen often has a reduceddynamic range compared to the original pixels, such as the pixels thatcome from a wide dynamic range digital camera (like the Thomson ViperFilmstream camera) or that come from a digital scan of camera negativefilm. However, digital video typically does not provide much extensionof dynamic range within the original image versus the presented masteredimage.

As an alternative to duplicating pixels for both wide range and mastered(i.e., narrow-range print-emulated) versions, it may also be useful toduplicate the pixels only for some frames of some scenes and/or regionswithin frames. Certain objects within certain scenes can benefit fromwide dynamic original information, and other objects in other scenescannot. Thus, the addition of selective frames or regions within frameshaving wide range pixels in addition to complete frames of mastered(narrow-range) pixels can provide all or nearly all of the usefulinformation available in the wide range pixels (by providing the widerange pixels only when they are likely to be useful).

An alternative hybrid method is also useful wherein wide range pixeldata together with one or more presentation transforms for presentationare conveyed (or stored) for most regions of frames and/or for mostframes, but wherein duplicate pixels are sent otherwise. In this way,pixels can be conveyed in some regions and/or some frames withoutrequiring transformations of wide range data.

Combinations and duplications of these various approaches are alsolikely to be useful for various purposes.

Spectral Information Available During Original Photography

Much of the practices of still photography and motion picturephotography are based upon alterations in scene colors duringphotography. For example, makeup is used to not only smooth and even theappearance of the skin, but it is also often designed to photographwell. Cameras and films are nearly always trichromatic (although a cyanfourth primary and other extra primaries are sometimes used). Film'sspectral sensitivity function, and the subsequent dye amounts, vary as afunction of exposure level. Digital cameras, however, usually have aconsistent spectral sensing function, and some cameras (like the ThomsonViper) can produce an output of the linear light levels seen in thesespectra. It is rare, however, that the spectral sensing functions arelinearly transformable into any color matching functions used formodeling human vision. The nearly ubiquitous alteration is that mostcameras and films sense a deeper wavelength of red than does humanvision. This results in artificially increased color saturation in somesituations, but also results in substantial color hue alterationscompared to vision.

Thus, it is useful to distinguish between photography as a creative andinterpretive tool, intended to create an appearance not matching thescene being photographed, versus use of photography to accurately andprecisely capture the colors of the image being photographed. There isalso a middle ground, wherein the scene appearance during originalphotography is mostly as intended, but may be altered somewhat duringmastering.

There are partial augmentations often available to the informationcontained in the trichromatic pixel values from cameras and digital filmscans. For example, the type of lighting is usually known, such as 3100KTungsten, Warm Fluorescent, Daylight at D55, etc. In the case offluorescent light, the mercury green wavelength line will be present,and this knowledge can be useful when attempting to understand theappearance of colors in the scene. Also, the camera's spectral sensingfunctions may be known. Further, the scene may contain a view of the skyat a known color temperature (such as blue sky at D90, which is a 9000Kequivalent having daylight spectrum).

Some matte (i.e., not shiny) objects may have known spectralreflectance, which can be used in conjunction with the lighting spectrato provide the spectrum seen for that object's surface.

Spectral Radiometry During Original Photography

A spectral radiometer can be used during original photography to measurethe spectral emission of various important objects and surfaces withinthe scene. Such information can augment the pixel values from the camerain defining original color.

It is also possible to imagine a high resolution moving image camerathat can capture the full spectral radiometry of every pixel in everyframe, plus a low resolution spectral measurement of the surround. Inthis way, the full characterization of the scene in front of the camera,or of a display or projection screen (if such a camera is aimed at adisplay or projection screen), is continuously being captured for everypixel for every frame. However, given that such a camera is notcurrently practical, we can begin by considering steps in thisdirection. For example, lower resolution spectral radiometry, and slowerframe rates, or measurements once per scene, can be used with presentequipment. We can also begin by adding additional spectral bands,equivalent to addition color primaries, wherein numerous spectral bandswould yield spectral radiometry in the limit, as the number of bandsincreases.

It is possible to adapt a spectral radiometer to make it into apractical scanning spectral radiometer. For example, FIG. 7 is a diagramof one embodiment of a scanning spectral radiometer. Aspectrally-neutral (or one accurately characterized for spectraltransmission) fiber-optic bundle 70, coupled to a spectral radiometer 71and to a diffuse light integrator (such as a frosted and/or milky glass,also spectrally neutral or characterized accurately for spectraltransmission) 72, can be used to create the sensing position within ablack light-tight box 74 (for example, a box lined in black velvet orthe like). A small square hole 76 can then be placed at some defineddistance from the sensing position defined by the integrating glass 72.The size of the square hole 76, the size of the integrating glass 72,and the distance between them, define the size of a “pixel” in thescanning spectral radiometer. The square hole 76 can be moved via linearactuators to scan across and down a scene (i.e., to scan as would pixelsin a camera).

On a fast spectral radiometer, such as one capable of measuring a normalbrightness scene in less than a second, an example scan size of 128pixels, configured as 16×8 pixels, can be scanned in about two minutes.If the “pixel” is two degrees (a common size for spectral radiometersusing lenses to focus on a spot), then the 16×8 pixels will span 32degrees by 16 degrees. The scene or set used during original photographycan be scanned by such a scanning spectral radiometer.

The amount of data in 128 spectral samples is negligible compared topixel data from still and moving image digital cameras. Such spectraldata can be considered as being a type of “meta-data” (which just meansdata that describes the scene being photographed).

Alternate configurations can be designed using lens elements and otherscanning mechanisms. For example, scanning could move the fiber bundle70 and integrating glass 72. Another configuration might use rotatingmirrors or other common mechanisms which are used for optically scanningan image.

Extending this design, a fast high resolution scanning spectralradiometer based upon this or other pixel scanning concepts can becreated which could capture higher resolution at faster speeds. Further,it is possible to design such a scanning spectral radiometer to fitright into the still or moving image camera, such that a reducedresolution version of the image can be continuously captured, at areduced frame rate, during original photography.

Several spectral scanning frames per second at tens of thousands ofpixels in resolution is still practical to convey. Further, there issignificant coherence in the spectral pixels, in that they will beregionally and temporally similar to each other, in the same way thatnormal (usually trichromatic) pixel values are regionally and temporallysimilar to each other. Thus, digital image compression can be usefullyapplied to spectral scan data pixels, as it is to normal (usuallytrichromatic) pixels.

Note that spectral radiometery scans can extend beyond the visiblewavelength range between 380 nm to 780 nm, and can potentially also seeultraviolet (near and optionally far) and infrared (near and optionallyfar). Such extra non-visible spectral energy has a variety of potentialuses, including helping to understand fluorescence as well as to helpidentify and define the materials and paints in the scenes (somematerials can be identified by their spectral signature).

Spectral radiometry scans can also be aimed at light sources within thescene, or be used with a fish-eye lens, or by taking multiple views viadifferent viewing locations and/or different viewing aims. A spatialspectral radiometric map can be made of all or part of the entireoriginal scene at the time of original photography. Note that a lightingsphere is sometimes used on movie sets to photograph a lambertian (i.e.,matte and diffuse) map of illumination, a specular (i.e., shiny) map ofillumination, and a reflective (i.e., mirror sphere) view of theillumination environment in the center of (or at other locations within)the scene. As an alternative or in addition, a panoramic photograph fromthe location can be made of the environment (with particular interest inthe lighting environment). These illumination maps have been used inconjunction with wide-range image capture to help re-light syntheticcomputer-generated simulated portions of the scene such that they matchthe photographed portions (these combined scenes are called ComputerGenerated Image, or CGI, composites). The present invention extends allof these concepts to the use of spectral radiometric pixel scans, suchthat not only illumination levels of the environment can be estimated,but the spectra of all illuminants can also be gained in the context ofthe scene environment. Such light spectra can augment a basic knowledgeof directional and diffuse ambient light location and/or direction(which can usually be inferred from the ball measurements) for use inunderstanding the spectra of objects within the scene. Anyspectrally-neutral gray object, for example, will usually reproduce thesum of the spectra of the illuminating light sources. As anotherexample, a round ball covered with fluorescent paint will fluoresce onits right side if there is a significant amount of ultra-violet lightcoming from one or more light sources on the right side of the scene.

Adaptive Spectral Mappings

The spectral mappings used for color matching functions, and theresulting color, can be varied adaptively to the circumstances ofpresentation and mastering in numerous ways, as follows:

-   -   The spectral and color mapping can be a function of pixel value,        pixel hue and saturation, etc.    -   The spectral and color mapping can vary regionally within the        image frame.    -   The spectral and color mapping can be specified to be a function        of extra data (like bit planes), which function can also        optionally adapt to viewers and the viewing environment and the        presentation display or projector+screen. Thus, the spectral and        color mapping can be a function of the presentation device, and        the presentation environment (e.g., ambient light and ambient        room color).    -   The spectral and color mapping can be a function of a single        specific viewer, or a group of specific viewers (even a        color-deficient viewer or viewers).    -   The spectral and color mapping can be a function of        world-regional preference. For example, some cultures like more        saturated colors, including face skin tones, other cultures like        colors less saturated, or like face tones to be more pink, or        more salmon, or more gray (for example). This is reflected in        the color processing preferences of national television systems        (and the associated electronic cameras, camera settings, and        telecine systems, and telecine adjustments), where the        television preferences of some nations use more saturated        colors, and some use less saturated colors, or different hues        for face skin tones, than the television preferences of other        nations.    -   The spectral and color mapping can be a function of preferred        color temperature for white (which may optionally also a        function of the color temperature of neutral in the ambient        surround).    -   The spectral and color mapping can be a function of the size of        the screen (either absolute, visual angle, or both).    -   The spectral and color mapping can be a function of image        presentation brightness level.    -   The spectral and color mapping can be a function of the size of        relatively-constant color regions (see histogram and        reduced-resolution methods described below).    -   The spectral and color mapping can be a function of the distance        away from the screen as well as of the intended center of view        (e.g., centering in the middle of the face of a speaking person        in a movie). Further, a likelihood that the eye will wander the        scene (a characteristic of some scenes and not others) can also        be considered.    -   Combinations of the above.

These concepts can be integrated into an overall conceptual andpractical framework for mastering and presenting accurate and precisecolor.

System and Method Overview

An overview of the system and method of the present invention is shownin FIGS. 8A-C. FIG. 8A shows an example of a mastering room 800 with aviewing screen 802 on which an image (an arbitrary shape, in thisexample) 804 is projected. Viewers within the room 800 may be seated atdifferent positions 806 at different angles to the screen 802. A numberof different mastering room characteristics will affect the display andperception of color on the viewing screen 802. For example, there may bered seats 806 or walls 808 in the mastering room, and light scatteredback from such surfaces will affect the colors on the screen. Thecontrast range of the displayed image 804 may also be affected due tothe presence of large bright areas on the screen, resulting in lightscattered back onto the screen 802 from the room. There may also belight spill coming from the projector lens or reflections from theprojector booth window glass (not shown), or from other sources, such asan “EXIT” sign 810 in the room 800.

FIG. 8C is a diagram showing the types and flow of data for accurate andprecise reproduction of color in accordance with the present invention.The spectral characteristics 820 of the image 804 as displayed on themastering room viewing screen 802 are measured and the spectral and/orcontrast confounds 822 are either measured (explicit) or determined(implicit, such as by measuring these characteristics once for the roomenvironment, and then assuming them to remain approximately constant, orby knowing characteristics of the materials or configuration of theroom). In addition, optional “secondary” environmental information 824may be measured or determined, such as a knowledge of where a person isbeing directed to be looking at the screen 802, the size of the screenin their field of view, each viewer's macular size and shape (or anaverage size and shape over a given population), an angle map, andmotion vectors (see the discussion of Eye Tracking and Viewing CenterProcessing below). The mastering pixel values (e.g., RGB or RGBW) 826for the displayed image 804, along with the corresponding spectralcharacteristics 820 and confounds 822 of the image and any optionalsecondary environmental information 824, may be stored (e.g., on film,tape or disk) or conveyed (e.g., by digital transmission), in originalor compressed form, as color image information 830.

To sum up, this aspect of the invention for the accurate and precisereproduction of color comprises measuring or determining the spectralcharacteristics 820 of the displayed mastering room image, measuring ordetermining the spectral and/or contrast confounds 822 of the masteringroom, optionally measuring or determining secondary scene information824, and conveying or storing the color pixel values 826 representingthe image plus the spectral characteristics of the image, the spectraland/or contrast confounds of the mastering room, and any optionalsecondary scene information. In one embodiment, the spectralcharacteristics of the image 820, the spectral and/or contrast confounds822 of the mastering room 800, and any optional secondary sceneinformation 824 are represented as a spectral map, with map framescorresponding to one or more image frames. Alternatively, each “class”of characteristics may be kept as separate data sets or mappings, sothat they may be selectively applied or omitted.

FIG. 8B shows an example of a presentation room 850 with a viewingscreen 852, such as a CRT or flat panel display or a projector andscreen combination, on which the image (an arbitrary shape, in thisexample) 854 from the mastering room source is displayed or projected.Viewers within the presentation room 850 may be seated at differentpositions 856 at different angles to the screen 852. In addition, theambient environment may have characteristics that affect the displayedimage, such as extraneous lights 858 or windows 860 (e.g., in a homesetting), or the room color and size (e.g., in a theater).

Again referring to FIG. 8C, in accordance with the teachings of thisinvention, the presentation spectral characteristics 832 of the image854 as displayed on the presentation viewing screen 852 are measured ordetermined, and the spectral and/or contrast confounds 834 of thepresentation room 850 are either measured or determined. In addition,optional “secondary” environmental information 836 may be measured ordetermined, such as a knowledge of the viewing characteristics and/orpreferences of the viewers 838, the size of the screen in their field ofview, each person's macular size and shape (or an average size and shapeover a given population), angle of view with respect to the screen 852,etc. In one embodiment, the spectral characteristics of the presentationimage 832, the spectral and/or contrast confounds 834 of thepresentation room 850, and any optional secondary scene information 836are represented as a spectral map, with map frames corresponding to oneor more image frames. Alternatively, each “class” of characteristics maybe kept as separate data sets or mappings, so that they may beselectively applied or omitted.

The presentation spectral characteristics 832 of the image 854 asdisplayed on the presentation viewing screen 852, the spectral and/orcontrast confounds 834 of the presentation environment, and any optionalsecondary environmental information 836 may then be combined 840 withthe conveyed (or replayed, from a stored form) color image information830, comprising Mastering pixel values and the corresponding spectralcharacteristics and confounds of the image and the optional sceneinformation, through appropriate color transforms. The transforms(discussed below) modify the color representation (i.e., the primaries)of the conveyed image to account for the measured and determinedcharacteristics of both the mastering room 800 environment (includingviewers) and the presentation room 850 environment (including viewers838) to more accurately and precisely convey the original, intendedcolors of the mastering room image 804 to the particular viewers of thedisplayed image 854 in the presentation room 850.

In essence, each display primary color is transformed (such as by linearor non-linear weighting) by application of the mastering room spectraland/or environmental information (such as in the form of a spectral map)and the presentation room spectral and/or environmental information(such as in the form of a spectral map). In some cases, presentationroom spectral and/or environmental information may not be available ornot used, and thus the transform will be based on the mastering roomspectral and/or environmental information conveyed or stored with theimage pixel data as a self-contained spectral mapping. Conversely, insome cases, mastering spectral and/or environmental information may notbe fully available or not used, and thus the transform may be basedsolely or primarily on the presentation room spectral and/orenvironmental information. For example, even if mastering room spectraland/or environmental information is unavailable, it may be useful totransform the image pixel values to particular adjusted color valuesbased on such presentation room factors as viewer age (to take intoaccount variance of color perception with age), viewing center (theportion of the image intended to be of most visual interest), color ofthe viewers' macula, and the size of the presentation image.

By deferring the binding and mapping of color representation onto apresentation device until actual use of such color representation,presentation-device independence is achieved without the need of adevice-independent intermediate representation.

Generalization of How a Color Gamut Relates to Spectral Mapping

The concept of a “color gamut” implies the use of a single set of colormatching functions. For example, the CIE 1931 chromaticity color gamutof a trichromatic RGB system is defined by the triangle connecting thechromaticity coordinates of each of the three primaries. Colors areconsidered “out of gamut” if they lie outside of the triangle (fortrichromatic linear systems) defined by the chromaticities of theprimaries. However, the gamut as defined according to CIE 1931 2 degreechromaticity differs from the gamut defined according to the CIE 1964 10degree chromaticity, since the x_bar2, y_bar2, z_bar2 color matchingfunctions of CIE 1931 differ significantly from the x_bar10, y_bar10,z_bar10 color matching functions of CIE 1964. Further, the CIE170-1:2006 modified colorimetric observer provides cone fundamentals toserve as spectral mappings for color matching functions, but the spectraof these cone fundamentals are a continuous function of viewer age (inthe range 20 yrs to 80 yrs) and angular width of the color being viewed(in the range 1 deg to 10 deg).

Thus, color gamut, as a concept based upon chromaticity, cannot beeasily generalized for use with the present invention, since the presentinvention generalizes the use of spectral mapping functions for use incolor matching for presentation.

In practice, a conversion of one color based upon a set of colorprimaries R1, G1, B1, each primary having an emission spectra, into asimilar color based upon another set of color primaries, say R2, G2, B2,each having a differing spectra from R1, G1, B1 respectively, involves amatrix transformation using integration of the emission energy spectrawith the spectral mappings of the color matching function(s) beingutilized for the transformation. The resulting values of each pixel interms of color primaries R2, G2, and B2 are “out of gamut” when any oneor more of their values (the amounts of R2, G2, B2, and/or otherprimaries, if more than three) goes negative. This concept for thelimits of gamut is not based upon chromaticity coordinates, and does notrequire a single set of tri-chromatic color matching functions (e.g., itdoes not require CIE 1931 x_bar, y_bar, z_bar). The color matchingfunction at every pixel may differ in the transformation between pixelcolors using R1, G1, B1, and pixel colors using R2, G2, B2. The gamutlimit can be defined independently as the range of positive values forthe amount of each primary for each pixel, even if each pixel utilizes adifferent set of color matching spectral functions.

It is often easiest to clip to the gamut by clipping negative R, G, B,(and/or other primary values if more than three) to 0.0, such that allvalues are 0.0 or above. However, a “soft clip” is also feasible,wherein the other primaries (two or more) may be altered when one ormore of the amounts of primaries nears 0.0. Thus, the slope of change ofthe one or more primaries, as they near zero, gradually smoothly movestoward zero as the pixel's color nears the gamut limit. Simultaneously,the other primaries can smoothly move toward their gamut limited values(usually a pure color, whether dark or light). The soft clip can becontrolled by the smallest primary (nearing zero) being liftedartificially in proportion to the amounts of the largest one or moreprimaries, as well as in proportion to the color purity (spectralnarrowness) of those one or more primaries.

Thus, whether hard-clipped to zero, or soft-clipped smoothly to zero,the use of zero for one or more primaries is the most useful concept inthe present invention to define the concept of the limits of colorgamut.

Hue and Saturation, LAB, and LUV Beyond CIE 1931

As with gamut, systems for hue and saturation, based either on LAB orLUV, also cannot be generalized, since both LAB and LUV are defined interms of CIE 1931 XYZ and/or chromaticity (xy). Basing LAB or LUV on CIE1964 does not provide much useful extension, since CIE 1931 and CIE 1964only represent two possible spectral mapping functions out of manyuseful such spectral mapping functions, for example, as defined incontinuous functions of age and viewing angle by CIE 170-1:2006.

Thus, a better concept for hue and saturation for use with the presentinvention is to define the ratios of one or more larger primaries to oneor more smaller primaries. Consider again the trichromatic example ofR2, G2, B2 spectral emission primaries, which could utilize a varyingcolor spectral mapping function when transforming from R1, G1, B1spectral emission primaries. The resulting R2, G2, and B2 may beevaluated with respect to each other, to form a variety of ratios, suchas R2/G2, R2/B2, R2/(G2+B2), R2/(R2+G2+B2), (R2+G2)/(R2+G2+B2). Suchratios and others can be useful in determining both hue and colorsaturation. Also, a variety of maximum and minimum functions and theirratios, such as Max(R2,G2,B2), combined with Max(R2,G2,B2) divided byMin(R2,G2,B2), can be useful in indicating color saturation and hue.Such formulae can also be generalized to more than three primaries. Suchformulae yield useful measures of hue and color saturation which areindependent of color matching functions (i.e., not dependent upon CIE1931 nor CIE 1964). Note that broad spectrum white and gray primarieswill be truncated differently, as will wide versus narrow colorprimaries when there are more than three primaries.

Another approach is to utilize a generalization of the types of hue andsaturation systems which are based upon LAB and LUV and CIE 1931, byutilizing a generalization of spectral color matching functions, such asCIE 170-1:2006, which is a function of age and viewing angle. In thisway, each pixel, or each region within a frame, and/or each sequence offrames, can potentially have an independent set of color matchingfunctions. The hue and saturation for a given pixel, or region ofpixels, can then be determined with respect to their spectral colormatching functions. For example, the white point spectrum of tungstenlight at 3200K is well known, as is the white point spectral definitionof D50, D55, D60, and D65 (not the correlated color temperaturedefinition which is based upon CIE 1931 color matching functions). Thesespectra can define a white point for any spectral color matchingfunctions by integrating those spectral color matching functions withthese one or more white point spectra. Hue can then be defined as adirection from that white point in the chromaticity space defined byeach such spectral color matching function, for any such spectral colormatching function, and saturation can be defined as a distance from thatwhite point. This need only be three dimensional, since the adaptivecolor spectral matching functions, such as those defined in CIE170-1:2006, although variable with respect to age and viewing angle, arestill three channels (long, medium, and short) for any given age andviewing angle. Further generalizations of the spectral color mappingfunctions, beyond CIE 170-1:2006, for example to adapt to specificindividuals, can also be utilized. This concept can also be generalizedto more than three primaries, including maximizing the broad-spectrumwhite (or gray) primary for the white point.

Note that scotopic vision, using the “rods” in the eye, does not conveycolor information, and thus may form a gradual desaturating factor atever lowering light levels. However, the use of this information differsbetween mastering and presentation. During mastering, low light levelswill be perceived as desaturated for a given spectra (usually composedof the sum of the spectra from three or more primaries). Duringpresentation, the color saturation may be artificially altered in themesopic region to increase or decrease saturation to adjust fordifferent viewers, different displays or projectors, different overallscene brightness on such displays or projectors, and/or differences inambient lighting surround. Such alteration may attempt to recreateaccurate color saturation (presented and/or perceived), or alternativelymay be used to intentionally increase or decrease saturation withrespect to that perceived during mastering (e.g., for a person whoperceives low brightness as having less color saturation than perceivedby average viewers of their age).

Note also that negative amounts of primaries (three or more) can bepreserved with various numeric representations, such as 32-bit floatingpoint, or signed integers, or OpenExr 16-bit half float data. Anyintermediate process which uses signed pixel values for primaries canretain negative values to preserve out-of-gamut colors. However, finalpresentation cannot make use of negative amounts of light via negativeamounts of primaries. An altered representation of the finalpresentation, by altering the ambient surround color, and/or by alteringthe white point of the presentation, may be able to bring negativevalues up to zero or to positive values.

Of course, altering the spectra of the three or more primaries, and/oraltering the color matching spectral functions being utilized for agiven pixel (or for a given frame or region within a frame) can affectwhether the amounts of each one or more of the primaries will remainpositive, or whether one or more would become negative.

Wavelength Precision of Spectral Color Matching Functions

Many color matching systems, such as CIE 1931 2 degree, CIE 1964 10degree, and CIE 170-1:2006, are defined at 5 nm intervals. The actualdata for CIE 1964 10 degree and CIE 170-1:2006 is the 1959 Stiles andBurch 10 degree data using 49 viewers, which is based upon wavenumber,which is a measure of spectral color represented as frequency, which isthe inverse of wavelength. The 5 nm data for all of these color matchingsystems are interpolated from the original wavenumber data, sinceuniform wavenumbers are different from uniform wavelengths. Datainterpolated to 1 nm, and even 0.1 nm, is available. However, there arevarious common interpolation algorithms to smooth and interpolate thedata. A smooth spline fit seems to be the most common method ofinterpolating between wavelengths. It is likely that data at 1 nm, orperhaps even finer, will eventually be needed. At present, much originalspectral color matching function data is only available at 5 nm, andeven that is often not justified by the original samples (such as theeven sparser wavenumber data upon which CIE 1931 is based).

Given that spectral radiometers often provide their data in 1 nm, 2 nm,and 4 nm intervals, and that color matching functions are oftenspecified in 5 nm intervals, the common precision at present is 1 nm.Thus, by interpolating all spectral emission and matching function datato 1 nm, integrals can be easily computed (via discrete integrals usingthe sum of multiplications) for use in spectral color matchingconversions.

For example, 5 nm color matching data, such as CIE 170-1:2006 data, canbe spline-interpolated to 1 nm, and 4 nm spectral radiometric data canbe spline interpolated to 1 nm. The resulting pair of 1 nm values can bemultiplied, and the results summed over all wavelengths between 380 nmand 780 nm to yield the required scalar values for use in linear colormatrix transformations. These matrix transformations, which are linearoperators, can then be applied to amounts of primaries expressed inlinear light (usually best expressed in floating point) in the form of amatrix (which will be 3×3 if trichromatic, such as RGB).

Non-Unique Mappings to More than Three Primaries

More than three primaries can offer an increased range of colors (calleda color gamut using specific color matching functions such as CIE 1931x_bar y_bar z_bar). Within the range of colors available for any threeprimaries of such a gamut, however, there are many possible combinationsof the four or more primaries which can result in a given coloraccording to any given set of color matching functions (e.g., for CIE1931, for CIE 1964, or for any of the range of color matching functionsavailable using CIE 170-1:2006).

A key principle of the present invention is that broad spectrum emittersshould be maximized whenever they are available. This principle,combined with the appropriate set of color matching functions, allows aunique optimization of four or more primaries.

If there is a fourth primary, it is recommended that the fourth primarybe a broad-spectrum white for mastering, presentation, or both.

All color matching functions are normalized to equal energy. This isequivalent to saying that the integration of the color matchingfunctions is equal, in energy units, with a uniform spectral energytheoretical white known as “E” (the equal energy white). Thus, an Euniform white will not vary with color matching functions, since theyare all normalized to be the same when applied to E. In human vision,all viewers will perceive an identical spectrum as matching itself.Thus, also, E will always match itself. If a white light source is asclose as possible to E in spectral energy, it will be reproducedaccurately between mastering and any subsequent presentation. See FIG.9, which is a graph showing the spectral output of common broad spectrum(white) radiators compared to the theoretical equal energy white (E).However, E does not exist in practice, and the most uniform broadspectral emitters are daylight (at around D60), tungsten (at 6000K,although usually much more yellow at 3200K), and Xenon (in theapproximate range of D50 to D65 using correlated color temperature,although Xenon has some spectral energy non-uniformity, as doesdaylight). Other broad spectral emitters also exist, but those justmentioned are the most commonly available broad spectrum white lightsources. Note also that UHP lamps are also being used in conjunctionwith liquid crystal projection (often retro-reflective), and DMDmicromirror projection (also often retro-reflective).

Thus the present invention recommends, if there are four or moreprimaries, that the fourth primary be broad spectrum white, in additionto red, green, and blue. See FIG. 10, which is a graph showing thespectral characteristics of one selection of RGB primaries and a broadspectrum white source (e.g., a high pressure 2 kW xenon arc lamp).

A narrow spectral emitter, such as a laser, will exhibit the highestdegree of variation with respect to variations between individuals, andvariations between color matching functions (e.g., variations due tochanging the age and viewing angle parameters of CIE 170-1:2006). Abroad spectral emitter will minimize differences between individuals aswell as differences of age and viewing angle, and will therefore be mostaccurate in reproducing color when this broad spectral primary ismaximized and all narrower primaries are minimized. Face skin tone haslow color saturation, and therefore could be most accurately reproducedfor all viewers under all viewing conditions using one or more broadspectrum primaries, while minimizing the narrower and more colorfulprimaries.

One can also conceive of levels of width and narrowness in normal red,green, and blue primaries, as well as with other useful primaries suchas deep red, yellow, cyan, and blue-violet. FIG. 11 is a graph showingan example of one narrow and one broad primary for each of red, green,and blue. When there are more than three primaries, the color range ofthe widest primaries should be utilized fully, and the narrowerprimaries should only be used at the edges of the color gamut (at themost colorful). For example, one could imagine three red primaries, onefairly broad and desaturated being a pink color, one a normal fairlybroad red similar to a red produced by a Xenon lamp and motion picturefilm yellow and magenta dyes together, and one a narrow red which isonly used for producing a saturated red or other saturated colors suchas saturated orange (e.g., combined with a narrow yellow) or saturatedmagenta (e.g., combined with a narrow blue-violet).

In addition to narrowing primaries about a relatively common peakwavelength, it will also sometimes be useful to have narrow wavelengthprimaries at slightly longer and slightly shorter wavelengths. Forexample, a broadband red primary having a peak at 690 nm might beaugmented by three narrowband red primaries at 730 nm, 690 nm, and 640nm. Utilizing this method will yield very saturated colors when thenarrow primaries are emphasized (with associated variations inperception of these narrow primaries between individuals), but mostcolors which are not as saturated will be mostly (or entirely) composedof the broad spectrum primaries (red in this example), and will minimizeinterpersonal variation and yield the highest accuracy and precision inconveying colors.

Utilizing the rule that broad spectrum primaries are always maximized,any number of primaries can be uniquely combined, under any variation ofcolor matching functions, to produce the most accurate color whichminimizes interpersonal variation, and minimizes variation due toviewing angle and personal age.

Note that the broadest spectrum primary should be optimized first, suchas broad-spectrum white (which will appear gray at low brightness). Thenthe wider-spectrum primaries (such as wide-spectrum desaturated red)should be optimized next, and then gradually the narrower primariesshould be utilized. For saturated colors, this process willautomatically yield almost no broad spectrum white nor broad spectrumprimaries. For normal colors, which make up the majority of most realscenes, the broad spectrum primaries, especially broad spectrum white,will be the dominant primaries.

Another useful approach is to use two medium-spectral-width primarieswhich smoothly sum to a wider primary when both are active together. Forexample, a fairly wide moderate saturation deep red with a peak at 700nm might pair with a fairly wide moderate saturation orangish-red with apeak at 600 nm to yield an effective broad spectrum red with a peak at650 nm. If dual greens and dual blues are similarly used, the rule wouldbe to utilize the maximum of these six (three paired) primaries bymaximizing whichever is the lower of the two primaries in each pairwhile still producing the intended color using the appropriate colormatching functions. See FIG. 12, which is a graph showing dual R, G, Bprimaries.

This concept of paired sub-primaries can be extended to matched groupsof three or four per red, green, and blue primary, as well ascontinuously extending across the spectrum including deep red, yellow,cyan, and blue-violet.

Any number of primary constructions are possible and practical with morethan three primaries using the optimizing principle of maximizing theresulting spectral breadth, and minimizing peaks (although peaks will benecessary, and will be large even though minimized, when creatingsaturated colors). Conceptually this is also equivalent to maximizingthe valleys, or low energy points, across the visible spectrum between440 nm and 730 nm (although long deep-red wavelengths from 730 nm to 780nm and short blue-violet wavelengths from 440 nm to 380 nm are notimportant to maximize, and may even be best to minimize due tosignificant interpersonal variation in these spectral ranges).

Note that such configurations can simulate color television primariesvia configurations of primaries which yield red, green, and blueprimaries which match those defined using CIE 1931 xy chromaticitypoints for each. This would be done using integration of spectralcombinations with CIE 1931 x_bar y_bar and z_bar and then to maximizeall broad spectrum primaries while obtaining the specified red, green,and blue primary x y chromaticities. The maximum breadth of each suchprimary can be utilized. However, it is also possible, even withtrichromatic television systems, to utilize broader spectrum primaries,including broad spectrum white, based solely on CIE 1931 xy chromaticityfor that pixel (as long as the pixel is not fully saturated at the edgeof the triangle bounded by the chromaticities of the red, green, andblue primaries).

Similarly, the “P3” primary chromaticities, as defined for DigitalCinema projectors (as a guideline for range of chromaticities) can besimulated for red, green and blue primaries. However, when CIE 1931 XYZ(with a 2.6 pure gamma) is utilized for Digital Cinema, then each CIE1931 xy chromaticity can be achieved using the maximum possible of thebroadest spectra, and only the minimum necessary narrow spectralprimaries to achieve a particular xy chromaticity.

With all systems based upon CIE 1931, however, there is no provision forinterpersonal variation, no variation for age, and no variation forviewing angle, thus limiting the potential accuracy and precision.However, the use of the broadest spectra to achieve any color willoptimize accuracy and precision, within the limitations due to havingcolors specified in terms of CIE 1931.

Normalization of Matrices

Use of CIE 170-1:2006 cone fundamentals as color matching functionsrequires normalization. As mentioned above, color matching functions arenormalized such that their integrals with equal-energy illuminant “E”are all identical, and usually set to 1.0. This is the same as sayingthat the area under each curve is equal, and is usually set to 1.0.

When integrating any color matching functions with actual display orprojector+screen emission spectra, it usually beneficial to applynormalization to the resulting matrix. This is true for all colormatching functions, including CIE 1931 x_bar y_bar z_bar, CIE 1964x_bar10 y_bar10 z_bar10, as well as all variations of cone fundamentalsl_bar m_bar s_bar from angular and age settings using CIE 170-1:2006.

Since the color matching functions are normalized to equal area, whichis usually 1.0, any additional normalization is effectively beingapplied to the scale of the spectral energy of the display orprojector+screen. It will usually be appropriate to set the display orprojector+screen to maximum white, with all primaries set to theirmaximum value (including when there are more than three primaries). Thiswill represent the “native white” for most displays and projectors, andwill usually be r=g=b=1.0 (or r=g=b=integer maximum, such as 255, 1023,4095, etc.). The native white may differ from the aesthetic white point,requiring that one or more of the primaries be reduced below its maximumvalue to create the brightest white that can be made at the aestheticwhite point.

Normalization of the mastering display or projector+screen can beperformed independently of normalization of the presentation display orprojector+screen. Conceptually, only one normalization is actuallytaking place, since mastering reference normalization and presentationnormalization are concatenated with each other in practice. Byconceiving of the normalization in this way, normalization should bedesigned using the maximum possible value of the mastering spectra(which is usually measured at the maximum for all primary values of themastering display or projector+screen) integrated with the colormatching functions, concatenated with the inverse matrix of thepresentation display or projector+screen spectra, again measured at themaximum value of the display or projector+screen. The resultingtransformation matrix should then be normalized to whicheverpresentation primary is largest for the presentation display orprojector+screen.

This procedure can be altered somewhat if the mastering display orprojector+screen is limited to only using pixel values which are limitedto the highest brightness of the aesthetic white. In this case, themaximum brightness aesthetic white proportions of the primaries can beutilized in the normalization process, using the maximum presentationwhite of the presentation display or projector+screen.

Note that this renormalization process will be required for everyvariation of color matching functions. Since the present inventionproposes optionally utilizing continuously varying color matchingfunctions, normalization must also be continuously performed. This willusually be implemented as a normalization maximum (of each primary'smaximum sum) and then a corresponding divide applied to the matrix. Thedivide can be implemented equivalently by a reciprocation, followed by amultiply of each matrix term, which is usually faster on most computers.Whether interpolating matrices over the image, or re-computing thematrices in a smoothly varying way across the image, the renormalizationwill be required for every pixel. If the affects of the changing colormatching functions on the resulting colors are determined to be smallerthan some threshold of perception, then a coarser renormalization can beused as a computation reduction. However, renormalization at every pixelis, in general, required when using color matching functions which varyover regions of the image.

Note that many cases of mastering and presentation spectra will besufficiently similar that normalization may be nearly constant. In thecase where mastering and presentation spectra are identical,normalization will be unity, and will not be affected by variations incolor matching functions. In the case where normalization is nearlyconstant, a range of color matching function variations can be tested,and whichever has the highest value for one or more primaries can beused for normalization. Variations in normalization when varying colormatching functions can then be ignored, and the single normalization canbe used for all matrices and all pixels over the entire image.

A determination of whether a single normalization can be utilized can bemade using the amount of variation in the normalization when varyingover the range of color matching functions which will be used in a givenpresentation circumstance. If the normalization maximum variation issmall, given the mastering and presentation spectra and circumstance(such as age settings and angle variations), then a constantnormalization can be utilized. Note that the maximum variation innormalization can be tested quickly and efficiently, and even performedonce per frame if necessary. Thus, each frame can be potentially testedover the range of color matching functions and circumstances of thatframe, to save computation with respect to renormalization (which may ingeneral be required at every pixel) if normalization variation for thatframe is small. Similarly, normalization variation can be tested withinregions of the image, or over multiple frames, shots, or scenes.However, per-pixel renormalization is efficient enough with currentdigital processors that these normalization optimizations will often beunnecessary.

At the other extreme, some combinations of mastering spectra,presentation spectra, and color matching function variation couldpotentially lead to moderately large changes in normalization. If suchlarge variations are determined for a frame or scene, it may bedesirable to limit the range of normalization. A significantly changingnormalization indicates a significant level shift in the reproduction ofmaximum mastering white (whether display or projector+screen white, oraesthetic white) over variations in color matching functions (given thepresentation spectra). It may be best to reduce this level shift (suchas scale the amount of change in normalization by half). If such casesof large normalization changes are encountered, it may be helpful togain intent or guidance about how best to handle renormalization fromthe manufacturer of the particular display or projector+screen, or fromindustry groups related to imaging standards or guidelines.

It is anticipated that large variations in normalization will not occurwith most normal mastering and presentation spectra combinations, undernormal variations in color matching functions as recommended by thepresent invention.

Conceptually, normalization is based upon reproducing the masteringdevice white or aesthetic white (if pixel values are limited in theirmaximum to maximum aesthetic white). In general, this will yield one ofthe presentation device primaries as being maximum, although it ispossible that more than one primary will be maximum. By varying thecolor matching functions over their entire useful range (such as varyingCIE 170-1:2006 between 20 years and 80 years in five year steps, witheach age further varying from 1 deg to 10 deg, in ½ degree steps), themaximum primary can then be determined for each such setting. It canalso be determined if the maximum primary switches to a differentprimary, or whether more than one primary are at their maximum at agiven setting. Given the maximum primary, the normalization is thendetermined.

Note that this normalization concept is based upon the maximum possiblebrightness of white available during mastering (whether device maximumwhite or aesthetic maximum white on that device) and the maximum whiteavailable during presentation. Intentional brightness alternations belowthese maxima during presentation are not considerations with respect tonormalization (although they could optionally be concatenated with thenormalization scale factor for implementation efficiency).

Note that this normalization methodology is the same whether there arethree or more than three primaries. However, this methodology requiresthat the maximum mastering display or projection+screen brightness ofwhite be set by having all primaries at maximum, or of aesthetic whiteby having at least one primary set at maximum. This methodology furtherdepends upon the maximum presentation display or projector+screen nativewhite, having all primaries set at maximum. Adjustments to alter thewhite at presentation versus mastering will usually be simplest ifapplied as scale factors subsequent to normalization. However, a higherwhite brightness can be achieved under conditions of altering the whitepoint if the alternation of the white point is included in thenormalization. In general, the alteration of white point is also afunction of color matching functions, which can vary over the imageaccording to the methods of the present invention. However, suchoptional white point alteration need only affect normalization if it isdesired to gain the incremental brightness increase which may then beavailable.

Hue Distinction

Using CIE 170-1:2006 cone fundamentals under variation of age andviewing angle (note that other cone fundamentals may also be used), itis possible (and novel) to use ratios of these cone fundamentals to showhue sensitivity at various wavelengths. This is shown at the top ofFIGS. 13A-13C through 16A-16C using the quantities:

-   -   long/(long+medium)    -   medium/(long+medium)    -   medium/(medium+short)    -   short/(medium+short)

In addition, the use of a weighting of these ratios by theircorresponding numerator cone fundamentals yields another useful measureof hue sensitivity. This is shown in the middle of each figure. Thisweighting makes each numerator be the square of its fundamental asfollows:

-   -   long²/(long+medium)    -   medium2/(long+medium)    -   medium2/(medium+short)    -   short2/(medium+short)

The greatest hue distinction sensitivities are at the highest slopes ofthese quantities. The greatest hue variation as a function of age andviewing angle can also be seen in the variations of these ratios. Theratio of absolute value of the delta numerator to the denominator isalso shown at the bottom, indicating the wavelengths having the greatestchange as follows:

-   -   abs(delta(long))/(long+medium)    -   abs(delta(medium))/(medium+short)

Note that the following have the same magnitude but opposite sign, thusthe absolute value of the delta ratio is identical:

-   -   abs(delta(medium))/(long+medium) [not shown, this is identical        to abs(dl)/(1+m)]    -   abs(delta(short))/(medium+short) [not shown, this is identical        to abs(dm)/(m+s)]

These various functions (described here and shown in FIGS. 13A-13Cthrough 16A-16C) can be used directly to compute hue distinction and huevariation as a function of age and viewing angle.

These functions can indicate hue sensitivity or variation for a givenwavelength or range of wavelengths.

Spectral Notches to Reduce Variation

Certain wavelengths vary more with viewing angle and age, and betweenpeople. Variation can be somewhat reduced by reducing the spectralenergy at one or more of these sensitive wavelengths. For example, anotch filter in the cyan at 490 nm with a width of ±15 nm (which shows alarge medium versus short cone fundamental variation with a person'sage) can reduce interpersonal variations at this sensitive wavelengthregion, and can reduce overall variation by a small amount. Othersensitive regions, such as 610 nm, 470 nm, and 420 nm, may also benefitfrom spectral notch filters. Such reduced spectral energy in sensitiveregions can slightly benefit both mastering and presentation, bysomewhat reducing interpersonal variation in both circumstances. Notethat some spectral regions, such as 550 nm (which varies significantlyas a function of angle), are very sensitive, yet this wavelength fallsin the center of the green primary, and thus energy reduction is notgenerally available.

Note also that the use of one or more such spectral emission notcheswill somewhat (usually slightly) reduce variation, independent ofwhether other aspects of the present invention are utilized.

Note further that spectral energy spikes or other increased energy atsensitive wavelengths during mastering or presentation will have theopposite affect, and will exaggerate interpersonal variation, as well asvariation as a function of viewing angle.

Flat Uniform Energy Regions to Reduce Variation

As can be seen from FIGS. 13A-13C and 14A-14C, there are wide spectralregions between 580 and 660 nm and between 420 and 550 nm which aresensitive to age. Further, as seen from FIGS. 15A-15C and 16A-16C, thereare wide spectral ranges from 510 nm to 650 nm and from 420 nm to 530 nmwhich are sensitive to viewing angle. Given that these bands representmost of the visible spectrum, only broad-spectrum white, as close to theE equal energy white as possible, is essentially invariant with respectto viewing angle and age.

As with broad-spectrum white, any flat uniform energy portion of anyprimary, including red, green, and blue primaries, will reducevariation. Variation is reduced when any primary is designed to berelatively flat, having relatively uniform energy, over any of thesensitive wavelength regions.

The principle of spectral notches at sensitive wavelengths, and ofrelatively flat regions at sensitive wavelengths, applies to red, green,and blue primaries as well as to optional additional primaries such asyellow, orange, cyan, yellow-green, etc.

Localized Wavelength Sensitivity

A useful way to determine the amount of spectral energy at variousnearby locations in a sensitive wavelength region is to integrate thespectral emission energy with two or more narrow spectral functions. Forexample, in order to determine the local wavelength variation in theamount of energy in the region near 520 nm, narrow spectral functionscould be used that are 4 nm wide at 524 nm, 520 nm, and 516 nm. Thedifference between the three integrated values would give an indicationof the amount of spectral emission energy change that can occur in thissensitive wavelength region due to changes in color matching functionsnear this wavelength (for example, as a function of age and/or viewingangle). Such tests can be performed at any number of critical wavelengthregions. This, in turn, can be used to guide accurate and precise colorprocessing by indicating the significance of the affect of age and/orviewing angle and/or other interpersonal variations in color matchingfunctions.

Application of Flat Regions and Spectral Notches to Color Paper Prints

The use of flat regions or spectral notches in regions of highsensitivity can be applied to color paper prints in addition to digitaldisplays and projectors. The paper, the dyes/inks, and the light sourcecan each utilize the principles of flat regions and spectral energyminimization in regions of high sensitivity to reduce inter-personviewing variation. Further, specific adjustments could be made to inks,paper, and/or light sources for individual color matching functions. Forexample, a specific paper could be selected based upon determinationthat such paper is better for a particular viewer. Similarly, differentinks or dyes could be selected for different people. Also, oneparticular light source, possibly with color balancing and spectralnotching filters, might be better for one person than for another. Thus,a characterization of a person's actual color matching functions couldbe utilized to select among choices for optimum presentation of color tothat person. In the absence of a specific person's characterization, orin addition to it, a general specification could be made based upon thesize of the image (e.g., the size of the color paper being printed, suchas 8″×10″), a typical viewing distance (e.g., 15″ from the eyes), and atypical position (and therefore brightness) of the light source, andpossibly light type (thus implying the spectrum) which is illuminatingthe color paper print. Further, the person's age can be used (such aswith CIE 170-1:2006 cone fundamentals as a function of age and/orviewing angle, used as color matching functions) instead of an actualpersonal characterization of the individual. Thus, many improvements inaccurate and precise color presentation are possible with color paperprints, in addition to displays and projectors+screens.

Spectrally-Adjustable Glass

It is also possible to use spectrally-adjustable transmissive opticalelements, such as glass or glasses (or any glass in the interveningpath), which can adapt to the viewer, and/or which can adapt to thesurround. For example, a spectrally-adjustable transmissive LiquidCrystal Display (LCD) panel may be used. Such an optical element can beapplied to viewing of paper prints, to light sources used to illuminatepaper prints, to displays, and to projectors+screens.

Energy Tradeoff when Using More than Three Primaries

If a display is made up of light emitting cells, each emitting aspecific spectral color of primary, then the amount of emission energyavailable for each primary will be the area proportion of the cells ofthat primary times the energy of that light emitting cell. For example,using triads and three primaries of red, green, and blue, each cellwould have a third of the area. The peak white would then be the fullred, green, and blue, each of which have a third of the area. Anothercommon pattern is a stripe of red, a stripe of green, and a stripe ofblue, each occupying one third of the area.

If there are two greens, one red, and one blue, in a Bayer patternsquare tile, then the maximum brightness peak white would be two greens,one red, and one blue. Note that this might result in a greenish white.

If one of the cells is white, one red, one green, and one blue in afour-tile, then peak white would be a quarter of the area of white, plusone quarter of the area for each of red, green, and blue. The maximumpure red is limited to a quarter of the area, whereas it could be onethird of the area (4/3 as much light) in the case of only threeprimaries of red, green, and blue having equal emission cell area. Thus,the maximum white can be traded off against the maximum brightness ofpure colors (typically red, yellow, orange, green, cyan, blue, andmagenta). Such configurations can be considered and enumerated in detailto optimize the presentation on a display having light emitting cells.

Color filter cells over a white backlight are also common, such as withLiquid Crystal Displays (LCD's). LCD's behave similarly to lightemitting cells in that the color primaries partition the light from thebacklight into independent areas (usually for each of red, green, andblue primaries, often in vertical stripes).

Other types of displays include Digital Micromirror Devices (DMD's),also called “micromirrors”, which modulate white light (usually having abroad spectrum) from a white lamp. The modulated light can be split intocolors using a color wheel. If this is done, there is a tradeoff betweena three-segment wheel of red, green, and blue, or a four segment wheelof white, red, green, and blue. The peak white will be increasedsomewhat in this configuration versus red, green, and blue in thirds.The same tradeoff exists as with adding a white light emitting cell tored, green, and blue light emitting cells, except that the time isdivided instead of the area.

A “three-chip” micromirror projector splits a white light source(usually having a broad spectrum) into red, green, and blue bands, whichare then modulated independently on the three chips, and then recombinedfor registered presentation on the screen. If a fourth channel is addedwherein some of the original white light is tapped off and modulated,this tapped-off white light is taken from the red, green, and blueprimaries.

In all of these cases, the production of a white reduces the maximumbrightness of saturated colors. Note that film, with yellow, cyan, andmagenta dyes, provides white with minimum deposition of dyes, thus beingclear film. The yellow, cyan, and magenta colors are single-dye colors,and thus have more brightness than red, green, and blue colors, whichrequire dye pairs (each of which absorbs some light). Thus, a similartradeoff exists with projected color film (either still or moving film).

This tradeoff of maximum saturated color brightness versus maximumbroad-spectrum white (assuming that the white can be broad spectrum) isa design parameter of any display or projector. The range ofbrightnesses, particularly the maximum brightnesses of saturated colors,becomes a design consideration when adding a broad spectrum white.

It is herein recommended that any fourth “primary” be broad-spectrumwhite, at whatever proportion of the light energy is considered best.For example, a given display configuration might select 30% of the peakwhite for the maximum broad spectrum white, where the peak white is thatwhite maximum plus the maximum of red, green, and blue primaries. Notethat a four-segment color wheel (white, red, green, and blue) with amicromirror is likely to yield a maximum of the white segment which isabout 65% of the white that results from also adding maximum red, green,and blue. Such a configuration may be convenient for practicalimplementation, even though 30% of peak white might be adequate for thebroad spectrum white “primary”.

By using a broad spectrum white as a fourth “primary” (up to the maximumamount of that broad-spectrum white), interpersonal-variations areminimized, and color accuracy and precision are optimized. Even abovethat white's maximum, wherein red, green, and blue primaries are added,the spectrum is still smoother than if it were to have been made solelyof red, green, and blue primaries, thus still providing an improvementin color accuracy and precision.

With more than four primaries (or more than three, if no broad spectrumwhite is used), additional colors can be added at yellow, cyan, deepred, and deep blue, as well as broad and narrow spectral versions of allcolor primaries (including red, green, and blue).

Similar tradeoffs exist for any number of primaries due to lightemission area, proportion of time on a color wheel, or dichroic colorband split proportions. Thus, the range to maximum white, and to maximumbrightness of saturated colors, is determined by the specificconfiguration of primaries, especially if there are more than three.

The guidance of the present invention is that broad spectral emittersare maximized at any required saturation using color matching functionspectral weightings which are properly optimized (as described in thepresent invention).

Color Appearance

The techniques and formulae of color appearance modeling are continuallyevolving. A solid model has been developed by the CIE in CIECAM02 (CIEColor Appearance Model 2002). This and most other models are based uponaverage viewers and do not take into account any viewing angle issues,nor do they take into account the age of viewers. The concepts of thepresent invention can likely be applied to color appearance models tosignificantly improve their accuracy and precision.

Further, it is potentially possible and useful to characterize eachperson's interpersonal variations (to the degree possible) with respectto color appearance modeling parameters. As with age and viewing angle,which are averages, an individual's specific parameters can provide animprovement. The same is likely true for color appearance modelingparameters.

The most fundamental ingredient of color appearance models is theirability to provide an adaptation to a change in white point. This isusually done via a transformation to cone fundamentals, which is nearlyalways (in published color appearance models) approximated using amatrix transformation from CIE1931 XYZ tristimulus values.

Cone fundamentals, such as from CIE170-1:2006 (when used, the specifiedCIE 170-1:2006 cone fundamentals are used as color matching functions),allow direct von-Kries white-balance for color adaptation to white, byscaling by the integration of white with the cone fundamentals, and thenmultiplying by its inverse (e.g., dividing all colors' integrations withcone fundamentals by the level of white integrated with the same conefundamentals). Note that this is adaptive to changing cone fundamentals(e.g., by varying them as a function of age and/or viewing angle and/orindividual person differences). Note also that the use of a matrixtransformation from CIE 1931 x_bar y_bar z_bar to cone fundamentals willgenerally be inaccurate, and that the integration of actual spectra withthe actual cone fundamentals will produce a consistently more accurateand precise result for color adaptation to white. Further, thecapability of CIE170-1:2006 to adapt to average age and viewing angleallows further improvement in von-Kries white-balance color adaptation.Further, according to the present invention, the cone fundamentals ofCIE170-1:2006 can change with angular distance from the center of view,such that the white-point adaptation can be variably applied at everypixel (since the cone fundamentals vary at every pixel).

Note that the changing white-point color adaptation model CAT02 usingnarrower cone fundamentals, including negative values, can be applied tothe spectra for performing white point color adaptation. However, CAT02is not defined as a function of viewing angle nor average age. It isanticipated that improvements upon the von-Kries cone-fundamental coloradaptation models for changing white point, such as CAT02, and theHunt-Pointer-Estevez model, might eventually be able to take intoaccount viewing angle and average age, and thus improve their precisionand accuracy. The use of a spectral methodology, as described in thepresent invention, provides a framework for conveniently adopting anysuch improvements, should they occur.

Another fundamental and significant color appearance parameter is thecolor of the surround. In more complex color appearance models, thesurround itself is divided into the proximate surround (i.e., closeproximity surround), the surround, and the background. It is possible todetermine an individual's variations, or an average of a known group ofindividuals, in color perception to a particular surround configuration.It is further possible to measure the current surround. In viewingenvironments, such as movie theaters, where the surround does notchange, the surround can be measured once, and that measurement appliedsubsequently. In home viewing, however, daytime and night-timeillumination is often gradually (or suddenly, in the case of turning ona room light) changing. It is possible to augment any display to have itlook in various directions at the room surround with variouscolorimetric sensors (e.g., positioning sensors at the display cornerslooking in 45 deg solid angles at various directions). In such a way,the current surround, including close proximity, medium surround, androom ambient background, can be continuously measured (or measuredfairly often, such as once per minute, but applying a gradual changefunction with the same duration for the adaptation, such as changinggradually over a minute).

Thus, another optional aspect of the present invention is theaugmentation of presentation displays and projectors with colorimetricand photometric sensors aimed in various directions around the room andaround the image screen to provide appropriate ambient brightness andcolor information for use in color appearance models.

Given all of the potential variations, it may be beneficial to have somemastering rooms have a variety of displays and ambient surrounds, and/ora highly adjustable display or projection and ambient environment. Usingsuch variations, the appearance may be adjusted to be as intended, andthe formula and/or parameters for achieving this appearance under thesealterations can be conveyed along with the images. Of course, inpractice, there are so many possible variations that it would bedifficult to check and possibly adjust them all. Thus, this process mustrely on color appearance models, which can approximate most of therequired adjustments. Some scenes, under some alterations of viewingconditions, can be checked and adjusted. Using this sparse sampling ofcorrections and adjustments, a model of the appropriate overalladjustments can be developed which can then be conveyed with the images.

In practice, composites and regional color modifications may be complexto convey in this manner, requiring that such scenes be conveyed withlinear light as seen in the mastering environment. Simplified formulaeand/or parameters for presentation under presentation viewing conditionswhich differ from mastering can be sent to guide such alterations.Alternatively, a generic set of formulae and/or standardized parameterscan be applied for appropriately altering the presentation to retainapproximate equivalence.

Adjustments to the Age of Viewer's Color Matching Functions

One application of the present invention will be to utilize a specificperson's age and/or color matching functions (which may vary from theaverage of their age). Also, a group average of specific individualcolor matching functions can be used, or the group's average age can beused to select a color matching function during presentation. However,it is also possible to adjust the apparent viewing age to differ from aperson's actual age or actual color matching functions. For example, ifa key person during mastering is 30 years old, but a given viewer is 45years old, a useful option is to allow selection of color matchingfunctions that have a different average age than 45 years. The affect ofsuch an arbitrary adjustment to color matching functions will bedependent upon the change is spectra between mastering and presentation.Some amount of experimentation would be needed under given conditions ofthe age of any given viewer(s), presentation spectra, the age of keymastering personnel, and mastering spectra, in order to determine auseful variation to color mastering function age during presentation.After some amount of experimentation and selection, an adjustment andselection trend may become apparent which could then be usedconsistently without further experimentation.

One useful test data point would be to apply the color matchingfunctions of one or more key mastering personnel (or a blend for severalsuch mastering personnel) for presentation, instead of using colormatching functions for the viewers of the presentation. This would thenattempt to duplicate the colors presented to one or more of the keypersonnel during mastering.

Such alterations are likely the subject of testing and experimentation,since no simple rules are feasible given the unpredictable affects ofdifferent mastering and presentation spectra on variations in colormatching functions. However, it is likely that some consistentalterations will be useful for some viewers under some viewingcircumstances.

The basic premise of the present invention, however, is to providetechniques to accurately and precisely reproduce the colors of themastering display, as they are seen by a viewer during presentation.Thus, it will often be better to adjust color appearance parameters(such as saturation, gamma, contrast, or white point) rather thanaltering color matching functions. When using such intentionalappearance adjustments, the resulting appearance alterations willdeviate away from accurate and precise color reproduction, but will beof an intentional and directly-controlled nature (such as increased ordecreased saturation based upon viewing preference). Such alterationsmay be best performed under guidance for such alterations provided bykey personnel during mastering. For example, a cinematographer mayindicate that it is the intention that color saturation can be decreasedsomewhat (e.g., as much as 10%) but not increased, or that contrast mayincreased somewhat (e.g., as much as 20%) or decreased by a smalleramount (e.g., as much as 5%). Note, however, that viewing surround andabsolute screen brightness significantly affect the appearance of theseaffects (i.e., as modeled by color appearance modeling).

Multiple Versions

In addition to a “final” version of a mastered image sequence, it mayalso be useful to retain and optionally convey altered versions forvarious purposes. For example, a separate version might be made forpresentation on a high brightness (i.e., wide dynamic range) display.Such a wide dynamic range image is likely to be prepared with a deeperblack and a higher effective gamma than a normal dynamic rangepresentation, in addition to possibly containing extra bright regionswhich were significantly less bright on a normal brightness display.

As another example, a separate version might be made for a smallerscreen than a larger screen. Such altered versions are commonplace withrespect to widescreen versus narrow-screen displays (narrow screenpresentations of widescreen shows are also called “pan and scan”versions). Such altered versions are also standard practice in thatdigital cinema releases are mastered differently from digital videoreleases (such as high-definition and standard-definition DVD orbroadcast masters). A digital cinema master is made for a highercontrast presentation (e.g., gamma 2.6 and 1500:1 black to white ratio)and a wider color gamut (i.e., more saturated red, green, and blue colorprimaries with a D60 typical white point) than does digital video (e.g.,gamma 2.2 and 400:1 black to white ratio with less saturated Rec709 redgreen and blue color primaries with a D65 typical white point). Thus,the practice of keeping different pixel sets, representing differentversions of image sequences, applied to different contrast ranges andcolor gamuts, is commonplace.

The practice of creating different pixels sets for different uses can beextended for other variations in presentation. If a single set of pixelscan be used, with variations in processing (such as a different gammatransform or a different color saturation level), then the resultingmaster could serve multiple purposes from a common pixel set. Even withdigital image compression, the pixels are the dominant largest portionof the data, so such repurposing would efficiently use storage spaceand/or bandwidth.

A hybrid of various specified transformations from a common pixel setfor some regions of the image and/or some frames in a show, can be mixedand matched with regions and/or frames with independent pixels. In thisaspect of the present invention, some of the compactness benefits areretained where common pixels can be used for multiple variations, whileretaining the complete freedom of variation available with independentpixel sets.

Film output concurrent with digital cinema release in current practiceis another potential application of common or partly common pixels withmostly independent transformations. Multiple film print types or brandsmight be used in different parts of the world (e.g., Kodak Brand Vision,Kodak Brand Premier, and Fuji brands of prints). Large screen release(such as Imax 65 mm film with image noise reduction and sharpening) mayoccur simultaneously with normal digital and 35 mm film release. A4096-horizontal resolution release may be concurrent with a2048-horizontal resolution release, although the 2048-horizontal reducedresolution can be defined via a transformation, and need not exist asindependent pixels. Similarly, a 1920-horizontal HDTV release can alsobe defined as a transformation (whereas present practice is to create aseparate master). A variation in white point can be defined as atransformation, as can a variation in dynamic range, maximum brightness,color saturation, presentation gamma, and many other useful anddesirable variations. Another significant example is guidance concerningoptimal presentation in a dark surround versus a high-ambient surroundversus low-ambient surround. Guidance from key personnel can provideinformation, or even restrictions, concerning the acceptability oravailability of various useful or desirable or possible adjustments.

If each of the cinematographer, colorist, and director used differentdisplays/projectors with differing spectra, then each will beindependently processed, and each spectra can be conveyed, identifyingwho it is that used the given spectral-attribute display or projector.

Colorfulness is a function of absolute brightness, as well as to somedegree the darkness of the surround. Intent may be conveyed by thecinematographer, colorist, and/or director, such that in one case thecolors might be the correct hue, but have as high of a saturation(colorfulness) as possible at the highest brightness possible. Inanother case, bright colors should be presented with approximately thesame colorfulness as when they were mastered, adjusted for absolutebrightness and/or surround during presentation.

Absolute mastering brightness is useful information, since it can becompared with presentation brightness. Further, there is some evidencethat larger screens (farther away) at an equivalent subtended angle areperceived as brighter than smaller screens (closer) at equivalentsubtended angle, and equivalent measured brightness. Thus, absolute sizeof the screen is potentially useful information in addition to subtendedviewing angle and absolute brightness.

The ICC color standard provides for metadata slots to convey “hints”about colorimetric intent. However, there is little or no guidance abouthow to use these hints, which are generally left up to each maker ofcolor paper printers to interpret (nearly always undocumented, even ifany use is made of the hints). However, conceptually certain hints canbe provided by the cinematographer, colorist, and/or director aboutcolorimetric intent. For example, it might be the intent to make a sceneas colorful as possible, within the limits of brightness and color gamutof a given display (although gamut usually refers to range ofchromaticity as defined in CIE 1931 or CIE 1964, but has beengeneralized herein). Another intent might be to preserve naturalappearing color, possibly together with information about the type ofscene (tungsten indoors, daylight outdoors on a sunny day, daylight hazybut panoramic, etc.). This might yield different absolute measures ofcolor saturation at different brightnesses in order to appearapproximately the same. There are known similar affects with respect tothe viewing surround, and a person's adaptation (morning versus eveningviewing, having been outdoors in bright sunlight just previously, etc.).It should be noted that such issues are generally called “colorappearance modeling”, which is still only a fairly approximate model,and is known to vary significantly between individuals. However, someamount of processing is certainly feasible for both absolutepresentation brightness and ambient surround. The intent may also be tobe as accurate as possible in measured color being presented,independent of absolute brightness and ambient surround, which is also ameaningful approach in some cases.

Another useful intent is whether or not it would be beneficial toattempt to present a show at the same absolute brightness as was usedduring mastering. It has been common for movie masters to be made on aCRT at 25 fl, for intended digital projection at 14 fl. Similarly, ithas been common for movies mastered at 14 fl using digital projection tobe viewed using CRT displays at 25 fl. While the projector would notnormally be able to increase brightness to 25 fl (although there aremany useful cases of high brightness projection), certainly the CRTpresentation could be darkened a little down to 14 fl (although the CRTimage will usually be smaller and have less viewing angle, which mightrequire some adjustment in order to have equivalent appearance andcolorfulness).

Similarly, with the color of the ambient surround, there can be aperceived alteration in the color of the image on the screen. Forexample, if a room has warm-colored (slightly yellowish or reddish)walls, and tungsten (warm) lighting, an image mastered in a coolermastering environment (e.g., D65 video surround) will appear veryblueish (cool). Similarly, if an image is mastered in tungsten surround,and is presented in a room filled with ambient daylight (and perhapslight blue walls), the image will appear very reddish yellow (warm) evenif it is intended to be slightly bluish or daylight colored. Note thatsome mastering is done in a dark surround, where the white-point of themastering display or projector becomes the dominant factor relative tooverall color sense. It is common practice to master to a given whitepoint, which is implicitly known by presentation displays andprojectors. However, if an exact measurement is made of the white point(and its spectra), then this provides more detail which can be conveyedto the presentation projector or display system. If the masteringambient surround color differs from the white point of the masteringdisplay or projector, then this will affect perception during mastering,as it will during presentation. It is also feasible to convey theambient surround used in mastering, preferably spectrally (althoughbasic CIE 1931 chromaticity, or other chromaticity, will usually beadequate). Thus, the color of the ambient surround is another optionalfactor in affecting the perceived presented color versus the absolutepresented color. Another approach is to specify the master as being madeonly for correct absolute color, and specifying the correct colorbalance of the surround (e.g., D65 viewing, or D50 viewing or 3200Kviewing), thus relying on those who set up the viewing environment toset it up to match this surround viewing color requirement. Similarly,the absolute brightness level of the surround, or a dark surround, canbe a significant perceptual factor in addition to the color of thesurround, and can be similarly handled.

While it usually will not be practical in many situations to control thebrightness of the ambient surround, the color of the surround could beadjusted by providing illuminated surround in the presentation displayor projector. It is possible and practical for an illuminated surroundto provide a known ambient color, or alter the existing ambient color tocreate a known ambient color (e.g., add cool blue light to an otherwisewarm red room). If the resulting ambient surround color is not the sameas the mastering ambient surround, and/or the white point of themastered show, it may still provide a partial adjustment in the correctdirection, then requiring less correction.

It may be beneficial to provide for a presentation mode which presentsthe image “as cinematographer and/or director intended”. It will also bebeneficial to provide information about whether the cinematographerand/or director allow customization. If customization or variation isallowed, the nature and range of variation can also be provided. Thevariation range can be accurately described using concepts andcomputations within the present invention.

Conveying Personal Color Perception Information

It is now common that key rings contain electronic transmitters withbuttons used to unlock and lock home security and car security systems.The concept of such devices can be extended such that such devices givea periodic transmission of appropriate personal information. If apresentation display or projector were equipped to receive suchinformation, individual color perception information could be providedautomatically. Use of signal strength, of GPS location (perhaps extendedfor high location accuracy and precision), or other methods can beextended to indicate a person's location when viewing an image, and toindicate a person's presence in a given room. Infrared technology, suchas is common with television remote controls, could also be used.

Such information can also be communicated via a connection to a remotecontrol or to a display or projector. For example, a computer card (suchas PCMCIA), a memory stick, or a digital disk carrying data (such as DVDor CD) can be entered into the remote control, the display, or theprojector to provide accurate and precise personal color perceptioninformation. The preference settings in a display or projector can alsobe used (e.g., a “settings” group of menus is often available intelevision sets).

Personal preferences can also be provided. For example, a given personmay like warmer (yellower) color temperatures, whereas another personmay like cooler (bluer) color temperatures for the white and gray pointof scenes. It is therefore useful to establish presentation settingspartially based upon personal preferences

Note that gamma and white point are controlled via display settingsmenus on some personal computers (although the actual gamma often doesnot match, or else the gamma varies with brightness, meaning that it isnot an actual gamma at all). Also, the white point is set in terms ofthe average display and in terms of CIE 1931 chromaticity.

It is possible and practical to characterize individual viewing colormatching functions. Such characterization can be done in the home or atthe office. It is feasible to include special emission color spectra(e.g., via an internal special projector) within a display/projector foruse in determining color matching functions for each individual. Suchdetermination can be made in the presence of a person's surround, ifdone in their own home under daytime, dusk, and nighttime viewingconditions, further helping to characterize not only their colormatching functions, but their perception of colors in their actualsurround (for use in color appearance modeling). Special displays orprojectors could also be set up in television and computer stores,wherein a person may obtain an up-to-date (since it gradually changeswith age) characterization of their personal spectral mapping functions(such as every five years after the age of 20). It is feasible to obtaininformation as often as every day, or even every hour or minute, toaccount for adaptation (e.g., coming in from outdoors). Visual personalcolor perception information could be gathered at a visit to the eyedoctor. A person could match spectrally-different patches using avariable sum of 3 or more primaries (e.g., red, green, and blue),thereby yielding specific color matching functions for that person. Notethat this is how CIE 1931 was originally developed.

It would also be useful to provide visual acuity in the personalinformation. With such information, it would be possible whenappropriate to boost high detail, based upon viewing distance, and basedupon visual acuity.

There are many personal preferences that can be usefully conveyed inaddition to information needed for accurate and precise presentation ofcolor and brightness. Some people like bright images, others like lessbrightness. Some people like flash frames, others do not. Levels ofcolor saturation, the level of contrast, the maximum image brightness,etc., can be part of personal preferences. Flare correction for peoplewith flare in their vision is also possible to a limited degree.Families are more likely to have similar color vision parameters, or aremore likely to wear glasses, so preferences for a family may be based onmeasured parameters for one member.

Some displays already adapt to ambient room light levels. Suchadaptation can take into account the persons preferences and measuredvisual parameters as well.

There are regional preferences for color appearance (and otherappearance parameters), which are probably cultural and not genetic.Personal preferences will automatically embody regional preferences,when those regional preferences are held by a particular person. Anexample of regional television color preference is the variation incamera color settings and makeup-color variation in different countriesaround the world for different kinds of shows (like talk shows andnews-person presentation versus drama or comedy).

Preferences can be carried with each person, and altered over time,using personal data means (e.g., a personal computer or data store).Preferences can be complex and context sensitive. For example, a personmay like calm appearance, low color saturation, low brightness, andminimal scene cuts, when winding-down for sleep. However, for familymovie viewing after dinner, or for sports viewing during a holidayafternoon, high brightness and accurate color may be desirable. Aperson's or group's moods of the moment also may affect preference, suchas the desire at a given time to watch a scary or suspenseful movie withappropriate moods, or to watch a romantic light comedy, or an artisticdrama.

There is thus a broad range of useful information, both characterizingtechnical data for accurate and precise color presentation, as well asdescribing preferences, which may be used for color modeling andpresentation.

Eye Tracking and Viewing Center Processing

The eye has a yellow macular pigment over the fovea. Each person'syellow pigment is a bit different in size and shape, but the averagesize and shape is known. The pigmented area is near round at about 1.5degrees of viewing angle. It rolls off gradually at its edges (i.e.,blends thinly to nothing at its edges). Using a knowledge of where aperson is being directed to be looking, and the size of the screen intheir field of view, each person's macular size and shape, or an averagesize and shape over a given population, can be taken into account whenselecting color matching functions to model the person's color sensing.The 2-deg and 10-deg color matching functions are just generalizationsof this characteristic. Having a more accurate 1-deg to 10-deg model ofthis, with lens-yellowing-age adjustment, such as CIE-170-1:2006, canimprove this model. However, a knowledge of where someone is beingdirected to look (using eye tracking tests, for example, or by knowingthat the eye is being directed by the director to a given person's faceor a given object in the scene) can improve the color sensing model. Theperceived color “nulls-out” the perceptual difference of the macularpigment in the visual cortex, so that a constant color is perceived asconstant, even though the l, m, and s cones are not sensing the sameamount of signals under the macular pigment as outside of it. Usingthese factors, new color matching functions for l, m, and s can be madebased upon design principles which expand upon those described inCIE-170-1:2006. For example, FIG. 17 is a diagram of an image 1700showing different angular values that may be selected for color matchingfunctions based on various viewing angles G for the image (see table atbottom of FIG. 17) and the viewing center. Here, the viewing center is aperson's head and upper torso 1702; the narrowest color matchingfunction A is selected for pixels nearest the viewing center. Forregions further away from the viewing center, such as a distant cloud1704 or distant mountains 1706, wider angled color matching functionsB-F may be selected.

When using eye tracking, the average and standard deviation for a groupa viewers can be used to determine how consistent the viewing center is.If eyes are wandering the scene (either intentionally bydirector/cinematographer intent, or unintentionally), then a relativelycommon viewing angle (e.g., 2 deg) should be used everywhere in thecolor matching functions. Even in this case, the angle parameter at theframe edges can be widened some, since eyes will never be pinned on aframe edge (left, right, top, bottom) within a created show. If eyes areconsistently looking at or near the viewing center (either intended orunintended, but usually intended), then the angle parameter of the colormatching functions can be gradually widened at angles away from theviewing center, as well as at the frame edges. In non-narrativeapplications, such as color video surveillance, then a consistent angle(e.g., 2 deg) can be used everywhere, or optionally the frame edge canhave a small gradual increase (e.g., 4 deg) within a small amount (e.g.,10% of screen height and width) of the frame edge.

If automatic eye tracking (average over a number of viewers) is used,the eye movements should be adjusted to change precisely at scene cuts.The data should also be heavily smoothed within each shot. An angle mapcan be used, which can be softened by blurring, or which may begenerated algorithmically based upon angular distance from the center ofview. The angle and center should change gradually over time, as well aswithin regions of each frame.

Motion vectors and confidence values can be used (see, for example, U.S.Patent Application No. 60/921,644 entitled Flowfield Motion CompensationFor Video Compression by the present inventor) to help determineinformation about a scene. For example, 1 deg viewing is not needed if aregion is moving 2 deg per frame. Thus, regions of high motion and highconfidence may wish to use 4 deg viewing, or even wider, since the pointof focus is devoid of sharp details (due to motion blur), and thus colorperception would naturally utilize a wider color region.

Motion vector/flowfield displacement length between frames can be usefulfor selecting the viewing angle parameter, since fast motion (largedisplacement length) implies larger color sense, because small detailcannot be focused during fast motion. Since motion vectors may becontinuous or discontinuous over regions of the image, their use withrespect to the angle of color matching function should be chosen tocorrespond to high motion using large angles, and low motion with astable viewing center using small angles for the color matchingfunctions.

Such an angular map (macular, non-macular) may be conveyed along withcompressed or uncompressed data as additional information, optionallyincluding center of viewing. The angular map can be scaled to anabsolute size in viewing angle during mastering for one or more of thedirector, cinematographer, and colorist. The angular map can also bescaled by actual presentation viewing angle (or an average or estimateif there will be multiple viewers) when used to adjust the angleparameter of color matching functions. Alternatively, the angular mapcan be created solely at presentation from no additional information,from the viewing center and presentation viewing angle (or estimate, ifmultiple viewers), or from other additional information, if available.Optionally, an additional function of the original viewing angle datamay also be conveyed, to indicate how it should be scaled (wide screenand wandering views, versus intense centered views), when the angularsize differs during viewing versus mastering. Whether or not the angularmap affects color is dependent upon the spectral changes duringpresentation versus during mastering. If the spectrum does not change(from mastering to presentation viewing, as in the days of theubiquitous CRT), then there will be no affect.

Encoders for image compression can usually afford more computation thandecoders. Pre-processing and analysis is thus possible in the encoder,with the resulting angle maps being sent for use by the decoder (as wellas having uses within the encoder).

The angle map can differ for each hue, but must change smoothly betweenhues. For example, if one hue covers a large region of a frame, then a10 deg color matching function would be most appropriate for that hue,even if there are small detail regions in other hues near the center ofview. See the discussion below concerning the use of histograms toidentify the regional extent of each hue.

The angle map can then use angular variations of the CIE 170-1:2006spectral color matching functions (which are called “cone fundamentals”in that document). There can be thousands of variations as a function ofage (20 years and below, up to old age, such as 80 years, continuously)and viewing angle (from 1 deg to 10 deg, continuously). Using the RGB(or more than three primaries) spectra against the appropriate age conefundamentals, the specific viewing angle used can be continuously variedper-pixel. While the specific color matching function can be altered foreach pixel, nearby and adjacent pixels should only vary a little andsmoothly. This can be accomplished by filtering (and therefore blurring)the picture down to one or more low resolutions (corresponding to one ora variety of viewing angles between 1 deg and 10 deg). If each pixel inthe low resolution image corresponds to two degrees of subtended viewingangle, then this would correspond roughly to 2-deg color matching. Theselow resolutions (for one or more of the variety of viewing angles) canthen be upsized back to full resolution to yield soft (blurred) images.The local color of each pixel can be compared to the soft images fromone or more of the viewing angles to determine how similar it is (viacolor differences which are insensitive to luminance variation, such asabs(u1−u2)+abs(v1−v2)). These difference channels can then be smoothedby filtering (and therefore blurring) them to low resolutions, and thenupsizing back to full resolution to yield soft (blurred) differencechannels. The difference channels can then indicate how similar eachcolor is to the color of its neighborhood for each of the candidateviewing angles. For example:

-   -   Start with the largest, e.g., 10 deg.    -   If similar, and thus small differences exist, use the 10-deg        color matching functions.    -   If not similar, check smaller sizes (e.g., 9 deg, 8 deg, etc.),        down to where the differences become small (if they do), e.g., 2        deg.    -   If the differences stay large, then use 1 deg color matching        functions.

Note that continuous functions must be used since color mappingfunctions must be smooth and continuous when applied to similar colorsand nearby pixels having those colors of the image. However, each pixelcan use a very different color matching function versus its neighbors ifthat pixel has a very different color versus its neighbors. It is onlynecessary that the colors to which the matching function is beingapplied vary continuously with respect to similar colors which arenearby.

Note that this results in a matrix for the components of each colormatching function (one matrix per viewing angle, for example). Thematrices can be locally parameterized using piecewise-linear orspline-fit or other smooth functions for each matrix term. This way, asmall number of color matching functions can still be smoothly applied(as long as the viewing angle blur is also smoothly determined) viainterpolating this small set of matrices with smooth curves (or evenwith piecewise linear).

Note that this process only affects color mastered with one spectra andpresented with another. If the same spectra is used to reproduce thecolors as was used to master them, this is a unity matrix for allparameters (e.g., for all viewing angles), since the integrations forthe RGB (or more than three primaries) source spectra will be equal tothe integrations with the same spectra for presentation.

An alternative to using CIE 170-1:2006 to vary angle is to graduallyinterpolate CIE 1931 2 deg x_bar y_bar z_bar into CIE 1964 10 degx_bar10 y_bar10 z_bar10, and use the continuously interpolated spectralmapping functions for continuously varying viewing angle. Other colormatching functions, both past and future, can also be utilized, as wellas color matching functions determined for specific individuals (orgroup averages).

Use of Regional Color Histograms for Determining Viewing Angle as aFunction of Color

The blurring method of discovering whether a given pixel's color iscommon to a wide angle of view in its surrounding has the disadvantagethat a small brightly-colored object having a widely-disparate colorwill move the average away from the predominant color, which mightotherwise match a given pixel. A solution for this in addition or as analternative to blurring is to utilize a color histogram for each regionof the image. A color histogram can be made luminance independent. Colormatching extent should be on the basis of hue and saturation, and not onbrightness, since an object of a given constant color might havelighting, texture, and shadows wherein luminance varies widely, but thecolor remains constant. For example, one could use u′ v′ based upon CIE1964 10-deg or u′ v′ based upon CIE 1931 2-deg color matching functions.One could also use one or more luminance-independent “opponent” colormodels based upon the CIE 170-1:2006 cone fundamentals.

In order to utilize a color histogram, the histogram must also be afunction of location within the image. For example, a low resolution(such as a resolution corresponding to 1 deg) down-filtered version ofnumerous histograms (e.g., 1000 histograms which span all active colors,and/or all possible colors) could be used. A given pixel's color couldthen be compared with the histogram buckets associated with its colorand with similar colors, using a function (such as linear rolloffweighting) of distance. If a large proportion of nearby pixels (such ashalf of them within a 10-degree region of ±5 degrees) fall intohistogram buckets having the same or similar colors, the location-basedhistogram would show that large numbers of nearby pixels might share agiven pixel's color (or a similar color). Thus, these pixels should use10-degree color matching functions. The choice of viewing angle for thechoice of color matching function should be a continuous function of theamount of the similar colored nearby pixels, and their nearness. Thenearness can be interpolated from the test pixel's location to yield anapproximately accurate distance to the histogram buckets of thelow-resolution histogram image. It is important that all functionsutilized be smoothly varying with both distance and color-similarity,including the fineness and interpolation methods related tootherwise-discrete histogram buckets.

The histogram method is computationally most efficient and smoothest ifthe current frame (or still frame) uses a fairly large memory. Thus, afairly high resolution histogram array, corresponding to regions of theimage should be used (for example, such as ⅛ of full resolution, bothhorizontally and vertically, having 1/64 as many pixels as the fullresolution image), with a moderate number (as large as is feasible tofit in memory) of buckets (for example, such as 32 hues and 32 levels ofsaturation, being 1024 hue/saturation bins). This example would require1024/64 or 16-times as much memory as the original image frame, which ispractical on today's computers. Linear or higher-order interpolationshould be used when comparing a given test pixel with both with thelocations and color similarity of nearby histogram locations, color, andhues.

The histogram method and the blur/low-resolution method can be usedseparately, or can be combined using min/max to see which is the bettermatch to a particular pixel's color, or by using a weighted average.Median filtering is also potentially helpful over small regions to findthe majority color in that region. Median filtering combined with smallregions and combined with hue and saturation ranges can find majoritycolors independently of brightnesses.

Novel Aspects

Novel aspects of the present invention include (but are not limited to)the following:

-   -   At least one implicit or explicit spectral interpretation of at        least one color primary which is not linearly transformable into        spectral mappings CIE 1931 x_bar, y_bar, nor z_bar nor CIE 1964        x_bar10 y_bar10 z_bar10, nor primaries (such as rgb) represented        as weighting in either of these two CIE spectral mappings.    -   Linearly or non-linearly weighting the amount of such spectral        mappings (e.g., pixels as OpenExr or other linear        representation, or non-linear pixels as video gamma, or pixels        as log or quasi-log, or other non-linear pixel representations).    -   Storing such information as one or more files for one or more        frames (not necessarily one to one files to frames), including        storing as compressed image data (still or moving).    -   Interpreting at least one primary color via a spectral mapping        (usually across 380 nm to 780 nm) which is self-contained with        the color data, where such data includes one or more image files        also containing pixels in one or more primaries (at least one of        which is interpreted via a self-contained spectral mapping), or        where such data is within compressed bitstreams and/or bitfiles,        decodeable into pixels which can be interpreted as a linear or        non-linear weighting of primaries (at least one of which is        interpreted via self-contained spectral mapping).    -   Mapping three or more primaries for image pixels (a primaries),        mapped into a+1 or more spectral weights (b weights) (e.g., 5 nm        or 1 nm bands), each weight then integrated with b−1 or fewer        sets (c sets) of m presentation and n perceptual weights,        resulting in c presentation primaries (where c is at least 3).    -   Multi-dimensional interpolation of spectral weights to account        for variations due to brightness during creation, and/or        intermediate processing, and/or mastering, and/or final        distribution presentation, including (but not limited to):        applying steeper gamma when using higher brightness displays;        applying darker blacks when using higher brightness display;        using knowledge of a particular display, or a particular model        or family of displays, and the associated spectral emission for        use with the spectra associated with the image, to yield an        optimized presentation for each color; taking into account the        surround/ambient viewing conditions; applying matrix transforms        by processing the spectra of the primaries, but not by        transforming the primaries themselves, and then re-integrating        these spectra with presentation primary spectra when needed        (this allows the number of bits (and number of bins) to be        minimized (keeping them above the noise floor) in order to        improve lossless compression, since there need be no cross-terms        (which greatly increase the number of lossless bins)).    -   When using more than three primaries, use of one or more broad        spectrum emitter channels in mastering and/or presentation,        maximizing the energy given the broad spectrum emitters.    -   Adjusting displayed colors based on personalized spectral vision        information (trichromat information), or on averaged (over a        group or subgroup) personal spectral vision information.        Personal spectral vision information may be pre-defined and        stored (e.g., in a credit card like device, or a storage device        carried on a person) so that it may be transmitted to a        particular display system.    -   Augmenting uniformly-sampled spectra with specific energy        spikes, each of specified wavelength and energy, removing their        energy from the uniformly-sampled spectra.    -   Augmenting pixel-values defined in mastering primary spectra        with object (reflective and/or luminous) or light spectra for        regions of the image. Similarly, augmenting with spectral        effects in such regions (such as fluorescence from UV        wavelengths to spectra and colors on fluorescent surfaces).

Programmed Embodiments

Some or all aspects of the invention may be implemented in hardware orsoftware, or a combination of both (e.g., programmable logic arrays).Unless otherwise specified, the algorithms included as part of theinvention are not inherently related to any particular computer or otherapparatus. In particular, various general purpose machines may be usedwith programs written in accordance with the teachings herein, or it maybe more convenient to construct more specialized apparatus (e.g.,integrated circuits) to perform particular functions. Thus, theinvention may be implemented in one or more computer programs executingon one or more programmable computer systems each comprising at leastone processor, at least one data storage system (which may includevolatile and non-volatile memory and/or storage elements), at least oneinput device or port, and at least one output device or port. Programcode is applied to input data to perform the functions described hereinand generate output information. The output information is applied toone or more output devices, in known fashion.

Each such program may be implemented in any desired computer language(including machine, assembly, or high level procedural, logical, orobject oriented programming languages) to communicate with a computersystem. In any case, the language may be a compiled or interpretedlanguage.

Each such computer program is preferably stored on or downloaded to astorage media or device (e.g., solid state memory or media, or magneticor optical media) readable by a general or special purpose programmablecomputer, for configuring and operating the computer when the storagemedia or device is read by the computer system to perform the proceduresdescribed herein. The inventive system may also be considered to beimplemented as a computer-readable storage medium, configured with acomputer program, where the storage medium so configured causes acomputer system to operate in a specific and predefined manner toperform the functions described herein.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention. Forexample, some of the steps described above may be order independent, andthus can be performed in an order different from that described.Accordingly, other embodiments are within the scope of the followingclaims.

I claim:
 1. A method for accurate and precise representation of colorfor a series of master images, each master image being representable bypixel values representing a plurality of color primaries, including: (a)displaying each master image on a source display in a sourceenvironment; (b) measuring, or determining from previous measurements,spectral characteristics corresponding to at least one portion of atleast one master image as such master image is displayed on the sourcedisplay in the source environment; and (c) conveying information,directly or after intermediate storage, to a transformation environment,the information including the color pixel values representing each suchmaster image plus the spectral characteristics corresponding to eachsuch master image, the conveyed information being suitable fortransforming the color pixel values of each such master image togenerate a corresponding presentation image for display in apresentation environment on a presentation display distinct from thesource display by applying the spectral characteristics corresponding tosuch master image to the color pixel values of such master image suchthat the resulting corresponding presentation image approximates thecolor of such master image when displayed on the source display in thesource environment.
 2. The method of claim 1, wherein the transformationenvironment is the presentation environment.
 3. The method of claim 1,further including: (a) assigning one or more rules to one or more masterimages, each of the one or more rules embodying intention informationrelating to a preferred presentation on the presentation display of theone or more master images after transformation to correspondingpresentation images; and (b) conveying the one or more rules, directlyor after intermediate storage, to the transformation environment in aform suitable for altering, in a pre-defined way, the presentation onthe presentation display of the one or more master images aftertransformation to corresponding presentation images.
 4. The method ofclaim 3, wherein the one or more rules embody intention informationrelating to at least one of presentation brightness, colorfulness,extended color gamut, extended dynamic range, ambient surroundbrightness, white point, presentation contrast, maximum black-to-whiteratio, viewing angle, or off-angle viewing.
 5. The method of claim 3,further including selecting and applying at least one of the one or morerules based on one of presentation context or viewer choice.
 6. Themethod of claim 1, further including conveying a set of color pixelvalues representing each such master image plus the spectralcharacteristics corresponding to each such master image, wherein eachset member comprises a version of the master image having at least onepresentation characteristic unique to that version.
 7. The method ofclaim 6, wherein at least some set members share common pixel values forsome regions of the corresponding master image.
 8. The method of claim6, further including generating at least one additional version of atleast one such master image from a set member, wherein the at least oneadditional version has at least one presentation characteristic uniqueto such additional version.
 9. The method of claim 6, further includingconveying intention information, directly or after intermediate storage,to the transformation environment, the intention information relating toa preferred presentation on the presentation display of at least one ofthe set members after transformation to corresponding presentationimages.
 10. The method of claim 1, further including generating multipleversions of each such master image from the conveyed information,wherein each version has at least one presentation characteristic uniqueto that version.
 11. The method of claim 1, further including: (a)measuring or determining spectral and/or contrast confounds in thepresentation environment of the presentation display that affectdisplayed presentation images; (b) determining and applying a correctionfor the spectral and/or contrast confounds to further transform thecolor pixel values of at least one master image as part of thegeneration of a presentation image corresponding to such at least onemaster image.
 12. The method of claim 1, further including compressingat least some of the information before conveyance.
 13. The method ofclaim 1, further including: (a) defining one or more regions of at leastone master image, each such region having a selected visual appearance;(b) specifying one or more rules for a desired presentation visualappearance for such one or more regions; (c) conveying the one or morerules, directly or after intermediate storage, to the transformationenvironment in a form suitable for augmenting, in a pre-defined way, thepresentation on the presentation display of the one or more masterimages after transformation to corresponding presentation images so asto approximate the desired presentation visual appearance specified bysuch rules.
 14. The method of claim 13, wherein the selected visualappearance includes one of resolution-altering effects, color-alteringeffects, brightness-altering effects, contrast-altering effects,fluorescence, iridescence, sparkles, and radiant objects.
 15. The methodof claim 1, wherein the spectral characteristics corresponding to eachsuch master image are conveyed distinct from the color pixel values. 16.The method of claim 1, wherein transforming the color pixel valuesincludes applying a color matching function that varies with each pixelas a function of the color of each pixel.
 17. The method of claim 1,wherein the step of transforming the color pixel values includesapplying a color matching function that varies with each pixel in aregionally-varying manner.
 18. The method of claim 1, further including:(a) measuring a black spectrum for at least one master image displayedon the source display in the source environment; and (b) conveying theblack spectrum, directly or after intermediate storage, to thetransformation environment in a form suitable for transforming the colorpixel values of each such master image.
 19. The method of claim 1,further including: (a) selecting a white point reference color; and (b)displaying selected master images on the source display in the sourceenvironment with a reference border having the color of the selectedwhite point reference color.
 20. The method of claim 1, furtherincluding: (a) selecting a neutral gray reference color; and (b)displaying selected master images on the source display in the sourceenvironment with a reference border having the color of the neutral grayreference color.
 21. The method of claim 1, further including: (a)characterizing the flare of the source display; (b) conveying the flarecharacterization to the transformation environment; (c) measuring theflare of the presentation display; and (d) transforming the color pixelvalues in the transformation environment as a function of the conveyedflare characterization of the source display and the measured flare ofthe presentation display to reduce the effects of the flare of thepresentation display.
 22. The method of claim 1, further including: (a)receiving personal color perception information associated with aviewer; (b) transforming the color pixel values of each master image asa function of the personal color perception information associated withthe viewer to generate a corresponding presentation image; and (c)displaying the presentation image to the viewer on the presentationdisplay.
 23. The method of claim 22, wherein the personal colorperception information is stored in a portable storage device.
 24. Themethod of claim 1, wherein the presentation display includesspectrally-adjustable transmissive optical elements.
 25. The method ofclaim 1, wherein the conveyed spectral characteristics corresponding toeach such master image are integrated with one or more selected colormatching functions and implemented in the form of one or morecorresponding transformation matrices.
 26. The method of claim 25,wherein a subset of the one or more transformation matrices isassociated with one or more color primary.
 27. The method of claim 25,wherein the one or more color matching functions are selected as afunction of one or more of: camera spectral sensing functions, effectiveviewing width angle of a color region of the master image, effectiveviewing width angle of a color region of the presentation image, therelationship of a color region to the viewing center of one or moreviewers of either the master image or the presentation image, the age ofsuch one or more viewers, the relative size of the angle of view of thesource display versus the angle of view of the presentation display, theabsolute brightness and color gamut of the presentation display, andinter-personal variation of color perception of such one or moreviewers.
 28. The method of claim 1, further including: (a) measuring ordetermining spectral characteristics of at least one presentation image;and (b) integrating the spectral characteristics of such presentationimage with one or more selected color matching functions to generate oneor more corresponding transformation matrices.
 29. The method of claim28, wherein a subset of the one or more transformation matrices isassociated with one or more color primary.
 30. The method of claim 28,wherein the one or more color matching functions are selected as afunction of one or more of: camera spectral sensing functions, effectiveviewing width angle of a color region of the master image, effectiveviewing width angle of a color region of the presentation image, therelationship of a color region to the viewing center of one or moreviewers of either the master image or the presentation image, the age ofsuch one or more viewers, the relative size of the angle of view of thesource display versus the angle of view of the presentation display, theabsolute brightness and color gamut of the presentation display, andinter-personal variation of color perception of such one or moreviewers.
 31. The method of claim 1, wherein the spectral characteristicsare represented as a spectral map, further including integrating two ormore spectral maps to generate one or more corresponding transformationmatrices.
 32. The method of claim 31, wherein at least one spectral maprepresents an associated color matching function.
 33. The method ofclaim 1, further including: (a) measuring or determining spectral and/orcontrast confounds in the environment of the at least one imagedisplayed on the source display; (b) determining a correction for thespectral and/or contrast confounds; and (c) applying a correction forthe spectral and/or contrast confounds to transform the color pixelvalues.
 34. The method of claim 1, further including: (a) measuring ordetermining secondary scene information for the environment of the atleast one image displayed on the source display; (b) determining acorrection for the secondary scene information; and (c) applying acorrection for the secondary scene information to transform the colorpixel values.
 35. A method for representation of color for a series ofwide dynamic range master images, each wide dynamic range master imagebeing representable by pixel values representing a plurality of colorprimaries, including: (a) displaying each wide dynamic range masterimage on a source display in a source environment; (b) applying atransformation to each wide dynamic range master image to define anarrow dynamic range master image; (c) defining color adjustments foreach narrow dynamic range master image; (d) conveying information,directly or after intermediate storage, to a presentation environment,the information including the defined color adjustments, each widedynamic range master image, and each narrow dynamic range master image,the conveyed information being suitable for generating a correspondingpresentation image for display in the presentation environment on apresentation display distinct from the source display.
 36. A method foraccurate and precise representation of color for a series of masterimages, each master image being representable by pixel valuesrepresenting a plurality of color primaries, including: (a) displayingeach master image on a source display in a source environment; (b)measuring, or determining from previous measurements, spectralcharacteristics corresponding to at least one portion of at least onemaster image displayed on the source display in the source environment;(c) determining a black spectrum for at least one master image displayedon the source display in the source environment; (d) conveyinginformation, directly or after intermediate storage, to a presentationsystem, the information including the color pixel values representingeach such master image, the spectral characteristics corresponding toeach such master image, and the black spectrum; (e) determining theblack spectrum of the presentation system; (f) correcting the colorpixel values representing each such master image using the conveyedspectral characteristics and the conveyed black spectrum to yield thespectrum of each color pixel value at the time of display of such masterimage on the source display in the source environment; (g) adjusting thecorrected color pixel values representing each such master image basedon the determined black spectrum of the presentation system to generatea corrected master image; and (h) mapping the corrected master image toa presentation system color space to utilize for presentation.
 37. Amethod for compensating for yellow macular pigment in the human eye inpresenting a series of images, each image being representable by colorpixel values representing a plurality of color primaries, including: (a)applying multiple color matching functions to transform each image, eachcolor matching function encompassing a different viewing angle andselected and applied as a function of the viewing center of a viewer andpresentation viewing angle with respect to display of the series ofimages on a presentation display, wherein the color matching functionencompassing the narrowest viewing angle is selected to most compensatefor yellow macular pigment in the human eye; and (b) displaying eachtransformed image on the presentation display.
 38. The method of claim37, wherein the viewing center is an average viewing center determinedby automatically tracking the eyes of one or more viewers with respectto a display of the series of images.
 39. The method of claim 37,wherein regions of each image farther away from the viewing center aretransformed by one or more color matching functions encompassing a widerviewing angle than the color matching function encompassing thenarrowest viewing angle.
 40. The method of claim 37, wherein colormatching functions are selected and applied separately for each colorprimary.
 41. A method for transforming a color image comprising pixelsrepresenting at least a primary colors, including: (a) for multiple setsof a primary colors, mapping the pixels of each set of a primary colorsto a corresponding set of b spectral weights, where b equals at leasta+1; and (b) for each set of b spectral weights, integrating such bspectral weights with corresponding c sets of m presentation weights andn perceptual weights measured in or determined from an environment inwhich the color image is displayed on a source display, where c equalsno more than b−1 but is at least 3, resulting in multiple sets of ctarget presentation primaries corresponding to the multiple sets of aprimary colors, where a, b, c, m, and n are integers.
 42. The method ofclaim 41, wherein the spectral weights are multi-dimensionallyinterpolated to account for variations in brightness of the pixels inthe color image arising from at least one of source display systemcharacteristics and source environment characteristics.
 43. The methodof claim 41, further including assigning a recommended presentationusage to each of the multiple sets of c target presentation primaries.44. The method of claim 43, further including utilizing the recommendedpresentation usage assignments to select a set of the c targetpresentation primaries for a specific presentation environment.
 45. Themethod of claim 41, further including selecting a set of the c targetpresentation primaries for a specific presentation environment based oncharacteristics of the specific presentation environment.